Your reader is synchonised so only one thread can use it at any one time. In me brain it's as though one thread reads a line, next thread reads a line, first thread reads a line, second thread reads a line.
In effect it's still really only one thread reading the file. There is another thing. Your synchronising access to your Array, you might not need to do that if your only doing reads, I helped a guy out recently and a well clued up guy suggested multi-threaded reads are usually ok, it's writes you need to watch for.
You could also see if you can speed things up by how your searching the array. But again it doesn't matter cos it works. Have to say I like the logic you have for creating enough threads to run one on each CPU.
Should have posted this link, it's to the post I referred to, there is a link to some threading information on MSDN Mag that might come in handy as well as some alternative, better performing, locking mechanisms.
Ask a question. Quick access. Search related threads. Remove From My Forums. Answered by:. Archived Forums. Visual Basic. Not for VB6 questions. Sign in to vote. Hi there I have a huge text file that needs to be processed. At the moment, I'm loading it into memory in small chunks x amount of lines and processing it that way. I'd like the process to be faster, so I'd like to try creating multiple threads, and having them load different chunks of the file at the same time and process it asynchronously.
Is it possible to do something like that, and if so, what would be needed to do so? Thursday, September 20, PM. New Post. Hi there I have a huge text file that needs to be processed. At the moment, I'm loading it into memory in small chunks x amount of lines and processing it that way. I'd like the process to be faster, so I'd like to try creating multiple threads, and having them load different chunks of the file at the same time and process it asynchronously.
Is it possible to do something like that, and if so, what would be needed to do so? Follow Post Reply. First, you have to devise a way for the multiple threads to know which lines to process.
If you are not going to delete from the file, then you still have to devise a way for each thread to know what to process. Trevor Benedict. Why not think about reading the file using a single thread and let other thread s process the chunks. This must be easier for you to deal with. I need the reading to be multi-threaded the most. Reading from the file is what takes the longest. Actually processing the chunks of the file takes much less time than the loading it into memory.
I may have figured it out, though Synchronized and also synchronized arraylists which were created using ArrayList. If the huge text file is on one hard drive and if the CPU processing is minimum as you have said in a separate post I don't see how multiple threads will help. Multiple threads are not going to make the hard drive spin any faster. C-Services Holland b. You're basically saying that the harddisk is the bottleneck.
It's supplying the data as fast as it can. Creating multiple threads reading that file would only make it slower because you'd be making the head of the drive jump all over the place to satisfy all the requests. Vaibhav Gargs. I like Optional 'thank-you' note:. In case we have to read huge files in GBs , then what is the optimal way to read the file using multithreading? Stephan van Hulst. Saloon Keeper.
Using more than one thread to read a file is usually a really bad idea. However, if the contents of the file were designed for it, you can do it. The file requires an index of sorts that says which records can be found in what position in the file. Read the index to find out what records the file contains, and divide them up over separate tasks that are responsible for processing a portion of the records. Why do you want to do this?
Thank you Stephan. It was on an interview question, so, just thinking how optimally we can read and process the large files. Campbell Ritchie. Won't the file be locked by the OS, preventing several threads accessing it in the first place? Consider reading line by line and passing the results to a parallel stream.
Operating systems at least Windows only lock files for a single process. The process itself can still access it with multiple threads.
0コメント