Mark Kruger posted comments about the performance of <CFDIRECTORY ACTION="list"> (see post 1 and post 2). He noted that returning full directory information with all file details (especially over network connections) can be painfully slow. As per his post, using Java APIs to return directory lists is far quicker, which is odd as <CFDIRECTORY ACTION="list"> actually uses the same Java APIs internally.
In a comment on Mark's blog, Tom Jordahl pointed out that the reason for this is that for <CFDIRECTORY> to return all file details it needs to first retrieve the file list and then perform a Java stat() for each file, and that stat() call is very time consuming (especially over network connections). The solution, if you just need file names and not other details, is to use the undocumented LISTINFO attribute. LISTINFO defaults to "all", but specify LISTINFO="name" and <CFDIRECTORY> will only return file names and will run just as quickly as the Java APIs it relies on.