I have code that reads ID3 tags from an mp3 file, but now I have some m4a files. I found some info on the structure of these files, but that doesn't mention ID3 tags.
What's the best resource for m4a file structure?
Is the song metadata in the m4a structure, or in the contained audio file (which appears to be AAC)?
M4A is just a filename extension - it still remains an MP4 container. Which consists of atoms/boxes (not chunks). The best resource is usually the documentation of the vendor himself, followed by experts with long experience, followed by additional details, followed by simplified explanations:
Apple: QuickTime File Format Specification; Metadata
ExifTool: QuickTime ItemList Tags
Multimedia.cx: QuickTime container; ยง 2.6: Meta data
xhelmboyx: MP4 layout = ISO 14496-1 Media Format
Just Solve the File Format Problem:
MP4, which uses the...
ISO Base Media File Format, which grew from...
QuickTime, which uses the...
Boxes/atoms format
Strictly by standard only the (MP4) container should have the overall metadata and any of the streams inside should not be searched for metadata. However, don't rely on this, and don't ignore potentially valuable metadata that can be in any/all of the streams (video, audio, subtitles, pictures...). Containers are like archives: they contain one or more files - and for each file you're back at where you began, because you have to recursively analyze that file again. AAC is by far not the only possible audio stream/codec - you could also run across an MP3 in an MP4 container.
ID3 can occur in MP4 as atom ID32, as mentioned here, but this is rare and only allows version 2.x, not version 1.
Additionally to the formats own metadata atoms other metadata formats (not specifically aiming at music) can be embedded in the following atoms:
system
atom UUID with value
other atoms
XMP
0xBE 7A CF CB 97 A9 42 E8 9C 71 99 94 91 E3 AF AC
XMP_ or xml or xmlc
Exif
0x05 37 cd ab 9d 0c 44 31 a7 2a fa 56 1f 2a 11 3e or JpgTiffExif->JP2
exif or exfc
IPTC
0x33 c7 a4 d2 b8 1d 47 23 a0 ba f1 a3 e0 97 ad 38 or 0x09 a1 4e 97 c0 b4 42 e0 be bf 36 df 6f 0c e3 6f
8BIM
0x2c 4c 01 00 85 04 40 b9 a0 3e 56 21 48 d6 df eb
360fly
0xef e1 58 9a bb 77 49 ef 80 95 27 75 9e b1 dc 6f
ID3 v2.x
ID32
Mostly the atoms in an MP4 have this layout:
- ftyp
- free
- mdat
+ moov
- mvhd
+ udta
- cprt
+ trak
- tkhd
+ udta
- cprt
+ edts
- elst
+ mdia
- mdhd
- hdlr
+ minf
- smhd
- hdlr
+ dinf
- dref
+ stbl
- stsd
- stts
- stsc
- stsz
- stco
+ meta
- hdlr (mdta)
- mhdr
+ keys
- mdta
- mdta
- mdta...
> ilst
+ (size, index)
- data (type, locale, value)
- itif
- name
- udta
> ctry
> lang
+ trak
https://docs.fileformat.com/audio/m4a/ has some details
https://github.com/ahyattdev/M4ATools has example code
The song meta data is in nested m4a chunks.
Related
I have a list that is <class 'bytes'> that is comprised of a 16-bit PCM value of <class 'int'>. The list is the result of a direct read of a segment of a 16-bit PCM wave file. I then create a numpy array from that built up list to save it as a separate wave file for training but wavfile.write() always fails because the 16-bit PCM data is wrong somehow, such as:
wavfile.write(savepath + 'wave_speechsegment_' + str(wavecnt) + '.wav', sr, nparray.astype(np.int16)) generates a ValueError: invalid literal for int() with base 10: b'z\xfe' error
And trying nparray directly: wavfile.write(savepath + 'wave_speechsegment_' + str(wavecnt) + '.wav', sr, nparray) I get ValueError: Unsupported data type '|S2
I try to set the list as 16-bit PCM values with:
hexval = struct.pack('<BB', val[0], val[1])
waveform.append(hexval)
nparray = np.array(waveform)
but when I save the 16-bit PCM values to the numpy file, python reports:
nparray is type: <class 'numpy.ndarray'> and nparray[0] is: b'z\xfe' and is type: <class 'numpy.bytes_'>
Saving to the numpy array segment to a file produces precisely the data set found for that segment in the source wave file, such as:
7A FE DE FE C5 FF 75 00 2F 01 76 01 99 01 55 01 05 01 74 00 05 00 9D FF 79 FF 65 FF 8C FF C9 FF
Can someone point me to information about how python deals with data, so that I can keep my 16-bit PCM data as 16-bit PCM data?
In [73]: astr = b'z\xfe'
In [74]: type(astr)
Out[74]: bytes
In [75]: len(astr)
Out[75]: 2 # 2 bytes
This is not a list. It's a string, more specifically a byte string, as opposed to the default (for Python 3) unicode string.
An array, created from such as string, will have a S dtype:
In [76]: arr= np.array(astr)
In [77]: arr
Out[77]: array(b'z\xfe', dtype='|S2')
In [78]: arr= np.array(astr+astr+astr) # + joins strings into one
In [79]: arr
Out[79]: array(b'z\xfez\xfez\xfe', dtype='|S6')
The data-buffer of the array contains those bytes. And can be view as other compatible dtypes.
In [87]: arr= np.array([astr+astr+astr])
In [88]: arr
Out[88]: array([b'z\xfez\xfez\xfe'], dtype='|S6')
In [89]: arr.view('S1')
Out[89]: array([b'z', b'\xfe', b'z', b'\xfe', b'z', b'\xfe'], dtype='|S1')
In [94]: arr.view('int16')
Out[94]: array([-390, -390, -390], dtype=int16)
In [95]: arr.view('uint16')
Out[95]: array([65146, 65146, 65146], dtype=uint16)
In [98]: arr.view('>i2')
Out[98]: array([31486, 31486, 31486], dtype=int16)
I'm an 'Old Timer' that learned to program on a Commodore 64 with a cassette drive (not a disk drive) for storing data. Oh the joy!
I am wondering if there is an equivalent way to perform Peek and Poke commands in a .bat file. Is it even possible anymore to check a specific address the way it worked in BASIC language?
Can a batch file locate the address of something like whether or not the 'y' key has been pressed and can it also set the value of that address to indicate that key was pressed?
It used to be something like PEEK(64324) would return the value of that location. Likewise; POKE(64324) would set the value at that location.
I could run a loop that basically waited for a keyboard input and if it recieved the correect trigger at that address it would perform a command. e.g.
For x = 1 to 1000
If PEEK(64324) = 1 then exit
Next x
So when the 'y' key was pressed, the loop would exit or goto the next command. Can BATCH check a specific address for it's current state and if so, is there any repository or listing somewhere that tells what address is what for things like colors and keys on the keyboard?
In MSDOS you can use the DEBUG tool to get a dump of memory:
SHOWBIOS.BAT
ECHO:d FE00:0000 0040 >debug.txt
ECHO:q >>debug.txt
DEBUG < debug.txt > debug.out
You can run the memory dump thru a script
-d FE00:0000 0040
FE00:0000 41 77 61 72 64 20 53 6F-66 74 77 61 72 65 49 42 Award SoftwareIB
FE00:0010 4D 20 43 4F 4D 50 41 54-49 42 4C 45 20 34 38 36 M COMPATIBLE 486
FE00:0020 20 42 49 4F 53 20 43 4F-50 59 52 49 47 48 54 20 BIOS COPYRIGHT
FE00:0030 41 77 61 72 64 20 53 6F-66 74 77 61 72 65 20 49 Award Software I
-q
Times have changed, indeed, but in fact you could perhaps still do PEEKs and POKEs with the good old Motorola 68k family... because they like the 6502 used memory-mapped I/O.
I could be wrong, but I think computers today largely have abandoned memory-mapped I/O. Instead they'll do something like the Intel 8x86 family. It's been awhile since I took 8086 assembly, though.
I have a huge dataset with the following structure
fieldA,fieldB,fieldC;fieldD|fieldE,FieldF;fieldG|fieldH,FieldI ...
where:
fieldA,fieldB and fieldC are strings that should be imported into separate columns
fieldD|fieldE,FieldF;fieldG|fieldH,FieldI is an array (elements separated by semicolon) of maps (elements separated by |) of arrays (elements separated by comma, e.g. fieldE,FieldF)
My problem is that the initial array is separated from the fieldA,fieldB,fieldC with a semicolon. My question is how do I set the separators correctly when I create a table.
This one does not recognize an array - although I provide a semicolon as a field separator
CREATE TABLE string_array(
first_part STRING # this would be to store fieldA,fieldB,fieldC
,second_part ARRAY<STRING> # this would be to store fieldD|fieldE,FieldF;fieldG|fieldH,FieldI and split it by semicolon
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '\\u003b'
COLLECTION ITEMS TERMINATED BY '\\u003b'
MAP KEYS TERMINATED BY '|'
STORED AS TEXTFILE;
LOAD DATA LOCAL INPATH '...' INTO TABLE string_array;
Any ideas how to make it work so I can build upon it? Thanks a lot in advance!
Great question.
I think that we can break this problem up into two discrete pieces: (1) Hive table structure, and (2) data delimiters.
Let's start by looking at the Hive table structure. If I understood your data structure correctly (please correct me if I didn't), the table structure that would best describe your data could be represented as:
CREATE TABLE string_array
AS
SELECT 'fieldA,fieldB,fieldC' AS first_part, array(map('fieldD', array('fieldE', 'FieldF')), map('fieldG', array('fieldH','FieldI'))) AS second_part;
Note that the field second_part is an array of maps, where the key to each map references an array of strings. In other words, the field second_part consists of an array within a map within an array.
If I use the statement above to create a table, I can then copy the resulting table to the local filesystem and look at how Hive assigns default delimiters to it. I know that you don't want to use default delimiters, but please bear with me here. The resulting table looks like this in its serialized on-disk representation:
00000000 66 69 65 6c 64 41 2c 66 69 65 6c 64 42 2c 66 69 |fieldA,fieldB,fi|
00000010 65 6c 64 43 01 66 69 65 6c 64 44 04 66 69 65 6c |eldC.fieldD.fiel|
00000020 64 45 05 46 69 65 6c 64 46 02 66 69 65 6c 64 47 |dE.FieldF.fieldG|
00000030 04 66 69 65 6c 64 48 05 46 69 65 6c 64 49 0a |.fieldH.FieldI.|
If we look at how Hive sees the delimiters we note that Hive actually sees five types or levels of delimiters:
delimiter 1 = x'01' (between fieldC & fieldD) -- between first_part and second_part
delimiter 2 = x'02' (between fieldF & fieldG) -- between the two maps in the array of maps
delimiter 3 = x'03' not used
delimiter 4 = x'04' (between fieldD & fieldE) -- between the key and the array of fields within the map
delimiter 5 = x'05' (between fieldE & fieldF) -- between the fields within the array within the map
And herein lies your problem. Current versions of Hive (as of 0.11.0) only allow you to override three levels of delimiters. But due to the levels of nesting within your data, Hive is seeing a requirement for greater than three levels of delimiters.
My suggestion would be to pre-process your data to use Hive's default delimiters. With this approach you should be able to load your data into Hive and reference it.
I have some Nikon raw files (.nef) which were rendered useless during a USB transfer. However, the size seems fine and only a handful of bits are shifted - by a value of -0x20 (hex) or -32 (dec).
Some of the files could be recovered later with another Computer from the same Card and now I am searching for a solution to recover the other >100 files, which have the same error.
Is there a regular pattern? The offsets seem to be in intervals of 0x800 (2048 in dec).
Differences between the two files
1. /_LXA9414.dump: 13.703.892 bytes
2. /_LXA9414_broken.dump: 13.703.892 bytes
Offsets: hexadec.
84C00: 23 03
13CC00: B1 91
2FA400: 72 52
370400: 25 05
4B9400: AE 8E
641400: 36 16
701400: FC DC
75B400: 27 07
925400: BE 9E
A04C00: A8 88
AC2400: 2F 0F
11 difference(s) found.
Here are more diffs from other files:
http://pastebin.com/9uB3Hx43
I am working on a C project in MSVS 2010 (meaning I am using malloc, calloc, and free, not the C++ new and delete operators). I need to find a memory leak(s?), so I've followed the steps on http://msdn.microsoft.com/en-us/library/x98tx3cf.aspx to get the program to dump the memory state at the end of the run.
I include the libraries like so:
#define _CRTDBG_MAP_ALLOC
#include <stdlib.h>
#include <crtdbg.h>
I also specify that every exit should display the debug info like so:
_CrtSetDbgFlag ( _CRTDBG_ALLOC_MEM_DF | _CRTDBG_LEAK_CHECK_DF );
But my debug output looks like this:
Detected memory leaks!
Dumping objects ->
{80181} normal block at 0x016B1D38, 12 bytes long.
Data: < 7 7 8 7 > 0C D5 37 00 14 A9 37 00 38 99 37 00
{80168} normal block at 0x016ACC20, 16 bytes long.
Data: < 7 H 7 X 7 \ 7 > A8 FB 37 00 48 E9 37 00 58 C2 37 00 5C AC 37 00
...
According to the article, I should be getting file name and line number output indicating where the leaked memory is allocated. Why is this not happening, and how can I fix it?
Adrian McCarthy commented that I should ensure that the definition _CRT_MAP_ALLOC existed in every compilation unit. While I could not figure out how to define that as a compiler option, I did create a sparse header file that I ensured every compiled file included. This made the debugging functionality work as expected.