All Questions
Tagged with performance filesystems
77
questions
0
votes
3
answers
2k
views
best filesystem for millions of files [closed]
Which Linux filesystem/setup would you choose for the best speed in the following scenario:
a few million files
~3mb file size on average
random access to files
need to get a list of all the files ...
- 31
1
vote
1
answer
469
views
Fragmented directory on ext4?
My ext4 filesystem loses performance when growing.
I have a system storing a lot of image files. This Debian based image server stores image files divided in year folders on 1-2TB disk sets with ...
- 31
3
votes
1
answer
1k
views
Confirming parameters for XFS filesystem and LVM volume striping over 2 ADAPT (RAID6-like) volumes
We are setting up an ADAPT0 (RAID-60-like) configuration for a file server.
We have six disk pools. Each consists of 14 disks and is set up using ADAPT. According to Dell's official white paper, ADAPT ...
- 209
1
vote
3
answers
2k
views
Setting recursive permissions: how to increase performance?
We have a huge directory tree, many levels deep and with a (very) large number of small files at each level.
From time to time we have large file data changes where parts of the tree are replaced and ...
- 4,756
8
votes
2
answers
19k
views
what is the relation between block size and IO?
I have been reading about disk recently which led me to 3 different doubts. And I am not able to link them together. Three different terms I am confused with are block size, IO and Performance.
I ...
- 233
0
votes
0
answers
121
views
File Server performance with simultaneous read/write operation
I have a WORKGROUP network with 50+ devices, all the computers can access a Windows Server 2008 R2 server which hosts all the media in the network. When one or some of the users are copying files from ...
- 153
0
votes
1
answer
930
views
What are all/most ways to enable/disable usage of async file-io in Linux?
To make a long story short, I have two identical servers hosting VMs using VirtualBox and both servers host one VM each nearly setup the same way, only that one is production and the other for ...
2
votes
1
answer
550
views
NTFS - writing small files performance variations
I have some test code that generates a list of 5000 small blocks of data (min. size: 351 bytes, max size: 386 bytes, average size: 385 bytes) and then writes them as separate files to a directory - ...
- 121
1
vote
1
answer
684
views
How to benchmark the performance of XFS with a large number of files?
I have a single server running some code that creates millions of small files all in a single directory. At first, the code runs extremely fast. After a day or so, it begins to slow down quite a bit. ...
- 42
3
votes
2
answers
4k
views
Poor performance when mounting Azure File Share on Linux VM
I have experimented with Azure File Service as a form of network file system that can be mounted by several VMs simultaneously -- something that regular Azure virtual hard disks are not capable of. An ...
- 33
2
votes
1
answer
241
views
How do you store VMs on HDDs with data integrity AND performance?
First off I know this topic easily gets subjective, but I'm trying to avoid that, as there should be at least 1 good answer to this in a sea of bad answers and it's hard to find.
At first sight my ...
- 389
4
votes
1
answer
587
views
Is it possible to create a software RAID 10 using SAN LUN's presented to server on 4 paths?
Is it possible to create a software RAID 10 using SAN LUN's presented to server on 4 paths?
I'm running Ubuntu 14.04 with 64 GB of RAM
Target is to achieve better I/O, storage and network teams have ...
- 87
32
votes
5
answers
12k
views
What are the performance implications for millions of files in a modern file system?
Let's say we're using ext4 (with dir_index enabled) to host around 3M files (with an average of 750KB size) and we need to decide what folder scheme we're going to use.
In the first solution, we ...
- 549
0
votes
1
answer
318
views
Performance comparison of single RAID vs multiple filesystems
I'm trying to get maximum sequential disk throughput for an application on my Linux server. It has 7 SATA disks that I could put in one RAID0 or RAID5 using a hardware RAID controller (HP P420i). Or, ...
- 267
12
votes
1
answer
16k
views
Sluggish performance on NTFS drive with large number of files
I am looking at this setup:
Windows Server 2012
1 TB NTFS drive, 4 KB clusters, ~90% full
~10M files stored in 10,000 folders = ~1,000 files/folder
Files mostly quite small < 50 KB
Virtual drive ...
- 293
4
votes
1
answer
6k
views
Filesystem Performance for LUKS Encrypted Volumes?
As you can format a LUKS volume with any filesystem you want, are certain filesystems better-suited to IO performance against an encrypted filesystem? Specifically, I'm wondering about a loop device ...
- 1,157
1
vote
1
answer
2k
views
What is the "right way" to combine sync and drop_caches?
When I need to do echo 3 > /proc/sys/vm/drop_caches with a sync beforehand, is it better to use
sync && echo 3 > /proc/sys/vm/drop_caches or
sync ; echo 3 > /proc/sys/vm/drop_caches....
- 38
5
votes
3
answers
3k
views
Generating a lot of dirty pages is blocking synchronous writes
We have processes doing background writes of big files. We would like those to have minimal impact on other processes.
Here is a test realised on SLES11 SP4. The server has massive memory, which ...
- 93
1
vote
1
answer
377
views
Slow file io performance after large network copy on Win7
In a step of our build process we run a tool that generates about 125,000 files. When generation is complete we copy them up to a network share so other systems can copy them down to process further.
...
- 44
3
votes
2
answers
5k
views
Optimize Linux file system for reading ~500M small files
We're seeing performance problems on a file system that contains ~500 M files that are relatively small (typically tens of kb) and ~150 K directories. Files access is predominantly reads and writes ...
- 131
0
votes
1
answer
865
views
OpenZFS on OSX filesystem dataset performance issue
tl;dr - My ZFS RAIDZ2 array reads at 7.5+ GB/s and writes at 2.0+ GB/s when I specify a bs=128K or greater with dd. OS X is assuming 1K (as per stat -f %k .) and all my is ~300MB/s; dd gives the same ...
- 101
6
votes
2
answers
733
views
Calling sync/fsync slows IO after 30 minutes uptime
After 30 minutes of uptime using Ubuntu 14.04 with an ext4 hybrid SSD I see many processes blocking IO using iotop.
The root cause of this slowdown has been traced back to the Unix system call sync.
...
- 141
2
votes
1
answer
382
views
Linux filesystem page cache - hot or not
How can you tell how much of the filesystem cache is hot (frequently accessed) or not?
The use case would be determining when you are approaching the limits of your memory but before you actually hit ...
- 21
5
votes
4
answers
776
views
Setting the noatime flag for filesystems inside an lxc container
My host filesystem is ext4 mounted with the noatime flag. But inside an lxc-container, the flag is absent.
Here's the output of mount on the host:
/dev/sdb1 on /home type ext4 (rw,noatime)
And here'...
- 225
6
votes
0
answers
180
views
Fast distributed filesystem for a large amounts of data with metadata in database [closed]
My project uses several processing machines and one storage machine. Currently storage organized with a MSSQL filetable shared folder. Every file in storage have some metadata in database.
Processing ...
5
votes
1
answer
3k
views
Filesystem performance degraded during RAID rebuilding
So quick question - our RAID6 array is currently rebuilding and there is a VERY noticeable filesystem performance hit (home directories are NFS mounted on the array).
I'd sort of expect that, given ...
- 471
32
votes
4
answers
34k
views
Performance difference between ramfs and tmpfs
I need to setup an in memory storage system for around 10 GB of data, consisting of many 100 kb single files(images). There will be lots of reads and fairly periodic writes(adding new files, deleting ...
- 1,831
0
votes
0
answers
968
views
AIX JFS2 I/O performance problems
I have a fresh installation of AIX 7.1. I noticed that JFS2 file system with logging enabled (log=/dev/hd8) is dramatically slower than one with log=NULL. Simple zip archive extraction takes 5 minutes ...
- 101
3
votes
2
answers
2k
views
FS for many small files, mostly reads?
I need a direction since all the blogs/benchs seems to contradict.
Goal:
Having a lot of small files on FS in a directory tree fashion similar to this AB/CD/EF...
Two size groups :
G1 : 10-20 kB
G2 : ...
- 145
17
votes
2
answers
3k
views
ext4 listing of files very slow in one specific directory that contained lots of files before
Background
I had a small logrotate misshap... Logrotate would rotate the archived logs by misstake causing a quadratic growth of files in my /var/log/. And by the time I caught wind that something ...
- 394
1
vote
4
answers
557
views
Slower file create on faster server. Why?
I have thwo servers. The old one is 24GB RAM, i7-950 @ 3,0Ghz, 2x 1,5TB SATA HDD (ST31500341AS). The new is 32GB RAM, i7-3770 @ 3,4GHz, 2x 3,0TB SATA HDD (ST3000DM001-9YN166).
Now I noiticed, ...
- 114
1
vote
1
answer
112
views
Is writing/reading speed affected by the name of the files?
How can I find out, or does anybody know, if is faster storing/reading a file with Spanish/Greek/Cyrilic/etc... characters like mi-foto-españa-oíóáaç.jpg than mi-foto-espana-oioaac.jpg ?
- 217
2
votes
1
answer
142
views
Why would disk reads be heavier than writes with a simple java app that creates many small files?
I've got a very simple java 7 app that reads from a proprietary file format (Hadoop sequencefile) sitting on disk xvdb and creates millions of 2-20kb files on disk xvdf.
ByteBuffer byteBuffer = ...
- 978
-1
votes
1
answer
262
views
Recommended resources for server tuning and security checklist for production Linux server deployment [closed]
There are many questions on server tuning and security checklist are being closed.
So, I am not asking for the exact detail.
Please point me to the resources you find it is helpful (no matter web ...
- 2,135
13
votes
2
answers
1k
views
Performance impact of running different filesystems on a single Linux server
The book "HBase: The definitive guide" states that
Installing different filesystems on a single server is not recommended.
This can have adverse effects on performance as the kernel may have to
...
- 7,939
-5
votes
1
answer
1k
views
CFQ vs Deadline - performance benefits [closed]
Running CPanel and wondering what would be the best for web hosting?
deadline is more balanced over CFQ right so would allow for more I/O activities right
1
vote
1
answer
1k
views
Linux user-space vs kernel -space file system performance
Is there a performance impact from using a user space file system vs a kernel space filesystem? I ask because I have a friend that refuses to use user space file systems because of a performance ...
user101130
1
vote
2
answers
11k
views
How much resources should backup tools be using? ( issues with clBackup )
My hosting provider makes backups of our servers using a tool called clBackup which pushes files over the network. Below is one ps output we took and reported to our provider:
top - 10:06:24 up 25 ...
- 3,523
11
votes
10
answers
4k
views
How does the number of subdirectories impact drive read / write performance on Linux?
I've got an EXT3 formatted drive on a Linux CentOS server. This is a web app data drive and contains a directory for every user account ( there are 25,000 users ). Each folder contains files that ...
- 927
3
votes
6
answers
20k
views
Performance gains of putting Windows pagefile on separate partition (same disk)?
I am making a drive image that will be deployed to several computers, and I'm tweaking it for performance. It is a Windows XP image.
Considering there is only one hard drive, will the machine gain ...
- 385
6
votes
5
answers
2k
views
Linux: how many disk I/O does it take to read a file? How to minimize it?
According to this paper on Facebook's Haystack:
"Because of how the NAS appliances manage directory metadata, placing
thousands of files in a directory was extremely inefficient as the
directory’...
- 3,110
1
vote
2
answers
262
views
Benchmarking disk performance on large-memory systems
I need to produce some comparative filesystem performance metrics for our virtual hosting environment. Specifically, I need to look at the difference between NFS and iSCSI and at the performance ...
- 43.9k
2
votes
4
answers
705
views
50,000 file in one directory, what is my best option?
I'm forced to have this directory structure
/var/www/$WEBSITE/$DIR1/$DIR2/$FILES
for each of those $FILES, there is approx 50,000 XHTML pages.
I'm running Cherokee, which has new front-end caching ...
- 2,471
1
vote
3
answers
462
views
ext2 dump/restore problem
I am running a mailserver with maildir storage. This means that quite a lot of files are created and I have just run out of inodes. AFAIK there is no magic command to increase number of inodes on ext# ...
- 173
12
votes
2
answers
5k
views
How does Linux handle concurrent disk IO?
When a Linux server is serving many concurrent requests to read many different files, does it:
Seek to File_1, read the entire file, then seek to File_2, read the entire file, then seek to File_3, ...
- 3,110
11
votes
4
answers
18k
views
How can I find out where a file is physically located on the disk (block numbers)?
This is an obscure question, I know. I'm trying to do some performance testing of some disks on a Linux box. I'm getting some inconsistent results, running the same test on the same disk. I know ...
- 917
0
votes
1
answer
316
views
Appropriate Linux (ubuntu 10.04) FS type for rapid FS management many small files and directories [duplicate]
Possible Duplicate:
Linux's best filesystem to work with 10000's of files without overloading the system I/O
I have an 240gb image store of approximately 1.5 million entries. About half ...
- 145
0
votes
1
answer
103
views
Store data on File Server or SQL database
We have a solution which includes storage of PDF documents.
Currently we have 50,000 documents that are 0.6 MB each (all documents same size)
We currently store these documents in varbinary(max) ...
- 2,229
3
votes
2
answers
207
views
Measuring disk seeks under windows
I would like to measure the number of disk seeks that are made by a process under Windows. If I open Task Manager, I can see a column named "I/O - reads". Is this the equivalent of disk seeks? Can I ...
- 447
1
vote
1
answer
195
views
Configure Windows filesystem for developer builds
I'm putting together a Windows box that will act as a continuous integration build server; we'll mostly build C# stuff using MSBuild, but I think this question applies to any compiled language on ...
- 497