C# on IIS Import of xlarge XML files .1-500mb + JPG images
$750-1500 AUD
Closed
Posted about 13 years ago
$750-1500 AUD
Paid on delivery
I have project where I need to import property data from XML feeds FTPd from an in-house server.
The XML feeds come in zip files, which need to be unpacked and also contain JPG images relevant to the property data.
The zip file size can be small to huge. Eg for the regular 15min updates the zip would be 1-2meg including images. The XML files would be 10-20kbs each.
When a full push of data is created the file size of the XML balloons out to well over 500mb (Just the XML) as there are 1000s of properties.
I require the XML file data to be inserted into a MSSQL Server DB, into 2 main tables residential_ & commercial_, along with 2 other tables resimages_ & commimages_ which will insert all the image file names and details.
The XML data is broken up into 3 parts, Res, Comm and Land, but I will supply the latest technical doc on the XML file structure.
It would obviously need to Insert or Update if the record is currently in the MSSQL db.
There also needs to be a check in place that XML files are not currently being uploaded before the script is actioned on the file to avoid cutting the file off and stopping the upload.
The website is hosted on latest IIS running Coldfusion 8 front end and custom CMS.
I require the all the images to be copied to a location in seperate propertyID folders as there are over 25,000 active images. This location also needs to be inserted with the image data.
While i'm not too familiar with machine code and C#, i've been told this is a great way import massive files like the XML at a fraction of the time it takes with CF or PHP etc. CF and PHP actually timeout when the file size reaches approx 3meg trying to import the XML.
If you require more information on the project to quote this kind of thing or would like to discuss, let me know.
Thanks.
I have experience importing xml data and images from zip files in a database. For the past few months I've been working on exactly this kind of projects. More details in PM.
Hi. I can create you Import application. I have 13 Years of experience in Software development Windows and Web. Expert in Sql Server and .NET. Microsoft Certified as MCP, MCTS, MCPD, MCT.
Yes, that's correct. CF and PHP have a limitation for processing the file with size above 3 MB as they are loaded into memory of web server where the web server its self also implements the maximum file size can be processed. Using C# you can access the remote file based on byte to byte method, so you don't need to worry about the time-out issue. The remote could in ftp or web server format, C# will be able to handle that.
Let me know the sample of XML file in zip format so I could review and make some estimation about the time-frame. Question: will it be a batch process so it should process any XML file within folders and sub-folders? But, it also checks if the file is locked by another process such as updating or progressing of something? Let me know.
Hello Sir,
We have 9+ years of development experience in .net technologies like C#.net, ASP.net, SQL Server. We are ready to start immediately.
Please check PMB.
Thanks,
VirtuosoIT
I am a C# developer. I have two years of experience in .NET development. I have good deal of experience in handling and parsing XML files using C# and I have ideas as to how we can make SQL insertions more efficient.
Regards.
Insiso Ltd has implemented similar import solutions for bulk XML and image combined exports, full details outlined in PM.
Feel free to contact me if you have any further questions or queries regarding our proposal.
I have 7+ years of experience in C# development. My specialty is XML importing to/exporting from MS SQL database. Many times synchronization and data imports were required for various (web) applications and sites redesign/upgrade.
Check PM.
I have more the 7 years of experience in C# and SQL Server I have done same kind of work in C# and SSIS.
I would say allow me to devolve this in SSIS.
I can do this for you in C# also.
Thanks