LFE is a collection of my C/C++ scripts I have used to edit and study very large files. These are loosely related simple scripts written at different times organized under one roof. Some scripts may be better optimized than the others. I continuously add new scripts to keep the LFE growing. This command line tool is currently available for both the 32-bit systems (Ubuntu 10.04, Debian, Windows) and 64-bit systems (Scientific Linux 6.1, RedHat, MacOS X). LFE is free software, the intention is to make it open-source in the future.
Some files are too large to be opened or manipulated by common unspecialized software such as MS Excel. Even special environments such as R may struggle with very large files since they may require that the entire dataset is read into the computer memory. These files typically need to be processed in small junks or by using special algorithms. This is were LFE comes in. However, the package also contains functions not specially designed for very large files. These functions are included for completeness sake and are there because I myself require them in my work and I either have not had time yet to make them faster or have not yet figured out how to do it. These functions are marked 'slow' in the online user guide. The functions that are likely not of interest to a wider group of users are marked 'specialized'.
The LFE has been written so as to be maximally modular. Each command/script performs a relatively narrow task. You can build more complicated processes by combinig the different modules linearly so that the outcome of one process becomes the input for another. For example you could modify the contents of some columns, followed by data extraction, followed by counting, followed by reformatting, followed by merging into the final file.
I assume no liability whatsoever for any part of the LFE or assume no responsibility whatsoever. The LFE is provided as is in every sense of the word. When you download and use the LFE you agree to do it completely at your own risk.
If you spot a bug, have a general question (either about running LFE or about the algorithms), drop me a line. I also take requests for incorporating new functionalities. Let me know what you are trying to do with large files and see if I can help you. If you would like to know if LFE can perform a certain process, or you want me to write a macro (sequence of commands) for you to perform a more complicated task with the existing functions, don't hesitate to e-mail me. Please send your e-mails concerning the LFE to this address: tom 'at' toomashaller 'dot' com, toomashaller 'at' gmail 'dot' com.