mkdir throws No space left on device , while creating a large file is fine( plenty space and inodes available ) - filesystems

Question

very strange behaviour
I cannot create a directory
[root#XXXXXX DEV]# mkdir 1
mkdir: cannot create directory `1': No space left on device
[root#dev-albert DEV]# pwd
/deployment/.octopus/Applications/OctopusServer/DEV
[root#XXXXXX DEV]# df -P /deployment
Filesystem 1024-blocks Used Available Capacity Mounted on
/dev/mapper/deploymentvg-deployment 10321208 5229888 4567096 54%
/deployment
[root#dev-albert DEV]# df -Pi /deployment
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/deploymentvg-deployment 655360 69129 586231 11%
/deployment
As you can see, plenty of space , good number of inodes free.
Does anyone have any clue what is happening with my system.
[root#dev-albert DEV]# dmsetup ls
rootvg-tmp (252:6)
rootvg-usr (252:7)
rootvg-var (252:8)
deploymentvg-usropenv (252:3)
deploymentvg-deployment (252:2)
rootvg-agent (252:4)
rootvg-oracle (252:11)
rootvg-varlock (252:9)
rootvg-deployment (252:5)
rootvg-swap (252:1)
rootvg-root (252:0)
rootvg-varspool (252:10)
top output
top - 14:44:35 up 347 days, 20:40, 2 users, load average: 2.02, 2.02, 2.05
Tasks: 125 total, 2 running, 123 sleeping, 0 stopped, 0 zombie
Cpu(s):100.0%us, 0.0%sy, 0.0%ni,117100.0%id,-42916200.0%wa, 0.0%hi,
0.0%si,200.0%st
Mem: 4071932k total, 3394132k used, 677800k free, 780312k buffers
Swap: 4194300k total, 22604k used, 4171696k free, 1742552k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
747 root 20 0 0 0 0 S 3.0 0.0 303:33.05 jbd2/dm-2-8
20679 root 20 0 0 0 0 S 2.7 0.0 0:14.24 kworker/0:2
16319 root 20 0 0 0 0 R 2.3 0.0 266:30.45 flush-252:2
When I run mkdir with strace
open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=99158576, ...}) = 0
mmap(NULL, 99158576, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fd4c4b04000
close(3) = 0
mkdir("1", 0777) = -1 ENOSPC (No space left on
device)
open("/usr/share/locale/locale.alias", O_RDONLY) = 3
write(2, ": No space left on device", 25: No space left on device) = 25
uname output
Linux 2.6.39-400.17.1.el6uek.x86_64 #1 SMP Fri Feb 22 18:16:18 PST 2013 x86_64 x86_64 x86_64 GNU/Linux

Answer 1

Sometimes that can be caused by the b-tree used by ext4 as directory index hitting its height limit. If you get the No space left on device error on mkdir for some names but not others, or there's plenty of space and inodes, check your dmesg for these warnings:
EXT4-fs warning (device dm-0): ext4_dx_add_entry:2226: Directory (ino: 80087286) index full, reach max htree level :2
EXT4-fs warning (device dm-0): ext4_dx_add_entry:2230: Large directory feature is not enabled on this filesystem
That means you're hitting the b-tree limit, and on a pinch you can enable large directory with:
tune2fs -O large_dir <dev>
It doesn't require unounting or rebooting, and it will increase the limit from 10M to 2B. Depending on what you're doing you're likely to hit performance bottlenecks or actually filling the disk before hitting the limit again, but I recommend rethinking your directory structure to avoid creating too many files and subdirectories in the same directory, and use the above solution only in an emergency.