Recently I had an issue while importing few thousand images and documents into the Sitecore Media Library. I had the following settings in Web.Config:
<!-- UPLOAD AS FILES Determines if media should be uploaded as files or as database blobs. Default: false --> <setting name="Media.UploadAsFiles" value="false" /> <!-- MEDIA - USE ITEM PATHS FOR URLS This setting controls if item paths are used for constructing media URLs. If false, short ids will be used. Default value: true --> <setting name="Media.UseItemPaths" value="true" />
Each time, it would import only a few files and it would stop. I tried killing all processes, lower the CPU and Memory usage but that didn’t make any difference.
According to the friendly and helpful Sitecore Customer Service Rep, Upload folder is managed by Sitecore.Resources.Media.UploadWatcher. Each time a file is added to the upload folder, UploadWatcher creates similar structure in media library. It is not a separate task or webservice, just simple uploading job started by “Created” event on filesystem.
The reason for this issue is that Sitecore uses System.IO.FileSystemWatcher object for file uploading (Sitecore use it to watch changes); when a lot of files are added at once, the FileSystemWatcher’s internalBuffer fills up and starts skipping files.
Checkout this article http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.aspx
As a result of my Ticket, they created a workaround which is available at http://sdn.sitecore.net/scrapbook/sitecore%20doesn%E2%80%99t%20upload%20files%20from%20upload%20folder.aspx. You would need a Sitecore developer account to login.
As a workaround you need to use Sitecore.Support.322918.dll file. The workaround increases the InternalBuffer to avoid the unexpected skipping of files.
1. Add the string <add type=”Sitecore.Support.UploadWatcher,Sitecore.Support.322918″ name=”SitecoreUploadWatcher”/> before each occurrence of the string <add type=”Sitecore.Resources.Media.UploadWatcher, Sitecore.Kernel” name=”SitecoreUploadWatcher”/>
(see <system.webServer>/<modules> section and <system.web>/<httpModules> section)
2. Comment out all such lines: <add type=”Sitecore.Resources.Media.UploadWatcher, Sitecore.Kernel” name=”SitecoreUploadWatcher”/>
3. Put the Sitecore.Support.322918.dll file into the /bin folder of your web site
4. In the web.config file, add the following string under the <settings> section
<setting name=”FileSystemWatcherBufferSize” value=”4096000″/>
Such value seems to be enough for 5000 files, but you can increase it further to be sure that even files with very long names are uploaded.
If the full file path contains 100 symbols, it takes 200 bytes, 32000 files will take 6400000 bytes (6250 kb), so the setting value should be:
<setting name=”FileSystemWatcherBufferSize” value=”6400000 “/>
As always, please backup before making any significant changes.
P.S.: UploadWatcher calls MediaManager.Creator.FileCreated(filePath); – you can try to fire the same method with path to the missing folder as parameter. There is another methos for FolderCreated as well.
Hope this helps. Credit goes to Sitecore and its Customer Service Reps.