Personal computing discussed
Moderators: renee, SecretSquirrel, just brew it!
While reader2.Read()
Try
Dim imageDownloaderThread As New SpawnDownloaderClass()
ThreadPool.QueueUserWorkItem(New WaitCallback(AddressOf SpawnDownloaderClass.downloadImage))
SpawnDownloaderClass.barline = reader2.GetString(0)
SpawnDownloaderClass.outside = reader2.GetString(1)
SpawnDownloaderClass.inside = reader2.GetString(2)
Catch ex As Exception
Dim logLocation = "\\SERVER\projects\FOLDER\errorLogs\" & Now.ToString & ".txt"
Dim errLog As New StreamWriter(logLocation, True)
errLog.WriteLine(ex.ToString)
errLog.Close()
End Try
Thread.Sleep(1000)
End While
just brew it! wrote:
Edit: I've done very little VB, so I may be mis-reading your code here. But it appears to me that you're using the same set of variables to pass the file name information to all of the threads (maybe this is the race condition you're referring to). This could explain why multiple threads seem to be trying to download the same file -- the variables are getting stepped on by the next iteration through the loop (when starting the next thread), before the thread spawned on the previous iteration has finished using them. You probably ought to be using the other form of QueueUserWorkItem() -- the one which allows additional data to be passed directly to the spawned thread.
I suspect there may also be some issues with the way you're doing exception handling; you probably should be handling exceptions in the spawned threads in addition to (or possibly instead of) in the parent thread. This means you'll also need to somehow coordinate access to the log file, or the error messages from the different threads may step on each other.
As I said, I haven't done much VB, so take the above with a grain of salt. (I've done *tons* of multi-threaded programming in C/C++ though, so I can generally spot the sort of race conditions that lead to weird behavior in multi-threaded apps...)
' pseudo-code
while myRecordset.Read 'loops and processes each record one at a time, forward only through the Database (sql)
create a thread and pass it the first record
another thead and second record
etc
end while
Now clearly what it will do instead and what I am pretty sure I am doing wrong at this point is that for each record, it's sending record 1 to each of the threads 1 time, and so on. So if I have 5 records, and 5 threads, 25 times. Surely wrong but I'm just working from a concept here.
just brew it! wrote:It is not uncommon for web servers to be configured to allow only 2 downloads at a time from the same IP address.
'create web request, read response as array of bytes, write to a file, close stream, close file
Dim request As HttpWebRequest = WebRequest.Create(strImage)
Dim response As HttpWebResponse = request.GetResponse
' -OR -
'I enabled this in an earlier test with the UI on so I could watch the downloads, all at once my downloads all pop up, but only 2 are downloads at a time.
My.Computer.Network.DownloadFile(requestImagePath, SaveToPath)
steelcity_ballin wrote:I would agree except that today due to the slowness we had been experiencing, we contacted someone in the know and they suggested exactly what I had just done, I don't see the benefit of all that extra programming for 1 extra download instance, seems kinda frivolous.
just brew it! wrote:steelcity_ballin wrote:I would agree except that today due to the slowness we had been experiencing, we contacted someone in the know and they suggested exactly what I had just done, I don't see the benefit of all that extra programming for 1 extra download instance, seems kinda frivolous.
Unless the person you contacted is very familiar with how this particular server was configured, he may not realize that there's a 2 stream limit.
I wouldn't worry about it. If you're getting any benefit from the multi-threading at all, you're probably already getting most of it with just the 2 threads. Adding more threads will probably result in diminishing returns, or even start to hurt.
steelcity_ballin wrote:I'm still hesitant to believe that the browsers have anything to do with this.
Flying Fox wrote:steelcity_ballin wrote:I'm still hesitant to believe that the browsers have anything to do with this.
It's not the browsers (client end) but on the server side.
Actually, the best way is to zip all these small files up and serve up that one bigger file (or 2). Is it possible?
BTW, your download program is a standalone GUI app, right? Can I see what you have now in terms of spawning the threads and controlling them?
just brew it! wrote:Well, at least now you won't waste any more time on it. And you learned a little bit about multi-threading and networking, so it wasn't a complete waste regardless.
just brew it! wrote:Depending on how sophisticated your router is, there may be some QoS (Quality of Service) settings which have been configured to prioritize the traffic.