Context
I'm implementing an integration application based on Apache Camel, which must interface to Google Pub/Sub infrastructure. For this purpose, I'm using the dedicated Camel PubSub component , which is based on the Google
PubSub HTTP Service API
For development and test phases, I would like to use the PubSub Emulator, and make it available on a shared DEV environment so that the whole DEV team can access it.
Issue
When running the emulator on localhost, everything works fine: the REST HTTP API is available (e.g. to create topics and subscriptions), and Camel route is consuming messages properly.
gcloud beta emulators pubsub start --host-port=localhost:8085
API Call result ( topic creation):
$ curl -X PUT -v http://localhost:8085/v1/projects/test-project/topics/topic-data
HTTP/1.1 200 OK
content-type: application/json
{ "name": "projects/test-project/topics/topic-data"}
But as soon as I start Emulator with -host parameter different than localhost (e.g. using my main IP address), the REST API does not work anymore and always return 503 UNAVAILABLE:
gcloud beta emulators pubsub start --host-port=[MY_IP_ADDRESS]:8085
Result when trying to create topic :
$ curl -X PUT -v http://[MY_IP_ADDRESS]:8085/v1/projects/test-project/topics/topic-data
HTTP/1.1 503 Service Unavailable
content-type: application/json
{"error":{"code":503,"message":"io exception","status":"UNAVAILABLE"}}
I have changed the emulator log level to FINEST, but nothing in logs can explain this behavior:
INFOS: Server started, listening on 8085
aout 16, 2019 11:43:51 AM io.netty.buffer.AbstractByteBuf <clinit>
PRECIS: -Dio.netty.buffer.checkAccessible: true
aout 16, 2019 11:43:51 AM io.netty.buffer.AbstractByteBuf <clinit>
PRECIS: -Dio.netty.buffer.checkBounds: true
aout 16, 2019 11:43:51 AM io.netty.util.ResourceLeakDetectorFactory$DefaultResourceLeakDetectorFactory newResourceLeakDetector
PRECIS: Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector#280b84c6
aout 16, 2019 11:43:51 AM io.gapi.emulators.grpc.GrpcServer$3 operationComplete
INFOS: Adding handler(s) to newly registered Channel.
aout 16, 2019 11:43:51 AM io.netty.handler.logging.LoggingHandler channelRegistered
PRECIS: [id: 0x369f8b68, L:/a.b.c.d:8085 - R:/a.b.c.d:49246] REGISTERED
aout 16, 2019 11:43:51 AM io.netty.handler.logging.LoggingHandler channelActive
PRECIS: [id: 0x369f8b68, L:/a.b.c.d:8085 - R:/a.b.c.d:49246] ACTIVE
aout 16, 2019 11:43:51 AM io.netty.util.Recycler <clinit>
PRECIS: -Dio.netty.recycler.maxCapacityPerThread: 4096
aout 16, 2019 11:43:51 AM io.netty.util.Recycler <clinit>
PRECIS: -Dio.netty.recycler.maxSharedCapacityFactor: 2
aout 16, 2019 11:43:51 AM io.netty.util.Recycler <clinit>
PRECIS: -Dio.netty.recycler.linkCapacity: 16
aout 16, 2019 11:43:51 AM io.netty.util.Recycler <clinit>
PRECIS: -Dio.netty.recycler.ratio: 8
aout 16, 2019 11:43:51 AM io.netty.handler.logging.LoggingHandler channelRead
PRECIS: [id: 0x369f8b68, L:/a.b.c.d:8085 - R:/a.b.c.d:49246] READ: 285B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 50 55 54 20 2f 76 31 2f 70 72 6f 6a 65 63 74 73 |PUT /v1/projects|
|00000010| 2f 74 65 73 74 2d 70 72 6f 6a 65 63 74 2f 74 6f |/test-project/to|
|00000020| 70 69 63 73 2f 74 6f 70 69 63 2d 64 61 74 61 20 |pics/topic-data |
|00000030| 48 54 54 50 2f 31 2e 31 0d 0a 55 73 65 72 2d 41 |HTTP/1.1..User-A|
|00000040| 67 65 6e 74 3a 20 50 6f 73 74 6d 61 6e 52 75 6e |gent: PostmanRun|
|00000050| 74 69 6d 65 2f 37 2e 31 35 2e 32 0d 0a 41 63 63 |time/7.15.2..Acc|
|00000060| 65 70 74 3a 20 2a 2f 2a 0d 0a 43 61 63 68 65 2d |ept: */*..Cache-|
|00000070| 43 6f 6e 74 72 6f 6c 3a 20 6e 6f 2d 63 61 63 68 |Control: no-cach|
|00000080| 65 0d 0a 50 6f 73 74 6d 61 6e 2d 54 6f 6b 65 6e |e..Postman-Token|
|00000090| 3a 20 66 31 61 38 33 63 35 36 2d 36 66 38 66 2d |: f1a83c56-6f8f-|
|000000a0| 34 65 35 32 2d 38 64 35 35 2d 64 64 66 66 64 37 |4e52-8d55-ddffd7|
|000000b0| 37 38 66 64 36 65 0d 0a 48 6f 73 74 3a 20 31 30 |78fd6e..Host: a.|
|000000c0| 2e 32 2e 34 31 2e 31 39 32 3a 38 30 38 35 0d 0a |.b.c.d:8085.....|
|000000d0| 41 63 63 65 70 74 2d 45 6e 63 6f 64 69 6e 67 3a |Accept-Encoding:|
|000000e0| 20 67 7a 69 70 2c 20 64 65 66 6c 61 74 65 0d 0a | gzip, deflate..|
|000000f0| 43 6f 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 |Content-Length: |
|00000100| 30 0d 0a 43 6f 6e 6e 65 63 74 69 6f 6e 3a 20 6b |0..Connection: k|
|00000110| 65 65 70 2d 61 6c 69 76 65 0d 0a 0d 0a |eep-alive.... |
+--------+-------------------------------------------------+----------------+
aout 16, 2019 11:43:51 AM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
INFOS: Detected non-HTTP/2 connection.
aout 16, 2019 11:43:51 AM io.netty.handler.logging.LoggingHandler channelReadComplete
PRECIS: [id: 0x369f8b68, L:/a.b.c.d:8085 - R:/a.b.c.d:49246] READ COMPLETE
aout 16, 2019 11:43:51 AM io.grpc.Context createStorage
PRECIS: Storage override doesn't exist. Using default
java.lang.ClassNotFoundException: io.grpc.override.ContextStorageOverride
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at io.grpc.Context.createStorage(Context.java:137)
at io.grpc.Context.storage(Context.java:129)
at io.grpc.Context.current(Context.java:181)
at io.grpc.Context$Key.get(Context.java:891)
at io.grpc.internal.CensusTracingModule$TracingClientInterceptor.interceptCall(CensusTracingModule.java:385)
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at io.grpc.internal.CensusStatsModule$StatsClientInterceptor.interceptCall(CensusStatsModule.java:691)
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at io.grpc.internal.ManagedChannelImpl.newCall(ManagedChannelImpl.java:819)
at io.grpc.internal.ForwardingManagedChannel.newCall(ForwardingManagedChannel.java:63)
at io.grpc.stub.MetadataUtils$HeaderAttachingClientInterceptor.interceptCall(MetadataUtils.java:74)
at io.grpc.ClientInterceptors$InterceptorChannel.newCall(ClientInterceptors.java:156)
at com.google.pubsub.v1.PublisherGrpc$PublisherStub.createTopic(PublisherGrpc.java:673)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.gapi.emulators.grpc.ProtoReflectionUtil.invoke(ProtoReflectionUtil.java:159)
at io.gapi.emulators.grpc.HttpAdapter$UnaryMethodHandler.handle(HttpAdapter.java:531)
at io.gapi.emulators.grpc.HttpAdapter.handleRequest(HttpAdapter.java:165)
at io.gapi.emulators.netty.HttpHandler.channelRead(HttpHandler.java:52)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:345)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:337)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:359)
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38)
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:350)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:495)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:905)
at java.lang.Thread.run(Thread.java:748)
aout 16, 2019 11:43:51 AM io.netty.handler.logging.LoggingHandler flush
PRECIS: [id: 0x369f8b68, L:/a.b.c.d:8085 - R:/a.b.c.d:49246] FLUSH
aout 16, 2019 11:43:51 AM io.grpc.internal.DnsNameResolver$Resolve resolveInternal
PRECIS: No TXT records found for localhost
aout 16, 2019 11:43:53 AM io.gapi.emulators.netty.HttpHandler$1 onError
INFOS: Exception when handling request: UNAVAILABLE: io exception
aout 16, 2019 11:43:53 AM io.netty.handler.logging.LoggingHandler write
PRECIS: [id: 0x369f8b68, L:/a.b.c.d:8085 - R:/a.b.c.d:49246] WRITE: 158B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 48 54 54 50 2f 31 2e 31 20 35 30 33 20 53 65 72 |HTTP/1.1 503 Ser|
|00000010| 76 69 63 65 20 55 6e 61 76 61 69 6c 61 62 6c 65 |vice Unavailable|
|00000020| 0d 0a 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3a 20 |..content-type: |
|00000030| 61 70 70 6c 69 63 61 74 69 6f 6e 2f 6a 73 6f 6e |application/json|
|00000040| 0d 0a 63 6f 6e 74 65 6e 74 2d 6c 65 6e 67 74 68 |..content-length|
|00000050| 3a 20 37 30 0d 0a 0d 0a 7b 22 65 72 72 6f 72 22 |: 70....{"error"|
|00000060| 3a 7b 22 63 6f 64 65 22 3a 35 30 33 2c 22 6d 65 |:{"code":503,"me|
|00000070| 73 73 61 67 65 22 3a 22 69 6f 20 65 78 63 65 70 |ssage":"io excep|
|00000080| 74 69 6f 6e 22 2c 22 73 74 61 74 75 73 22 3a 22 |tion","status":"|
|00000090| 55 4e 41 56 41 49 4c 41 42 4c 45 22 7d 7d |UNAVAILABLE"}} |
+--------+-------------------------------------------------+----------------+
aout 16, 2019 11:43:53 AM io.netty.handler.logging.LoggingHandler flush
PRECIS: [id: 0x369f8b68, L:/a.b.c.d:8085 - R:/a.b.c.d:49246] FLUSH
(note that message No TXT records found for localhost is also present when running emulator on localhost, so I'm not sure if this can be the cause)
Note the gRPC API is working fine in both scenario (localhost and specific IP address) but unfortunately I cannot use this protocole as Camel PubSub component is not supporting it.
I got the same result
{"error":{"code":503,"message":"io exception","status":"UNAVAILABLE"}}
but it started to work after I changed binding host from my external IP to global 0.0.0.0 host
gcloud beta emulators pubsub start --host-port=0.0.0.0:8085
I think i faced same kinda issue in past, difference is i was doing something different but whenever i was using my ip in url, url was not reachable.
Thing was some times when we are working with local system you will have to tweak your firewall in order to let your request get pass. It worked like a charm for me. I admit that all i did was disable my firewall but at the moment security was not that important to me.
Hope this might help. Please Do post if you find out if there was any other issue, I am very curious about this since i have to work with pubsub services very often.
Thanks.
Related
If I analyse multiple PDF files with a hex editor, I see that all of them have two trailers.
That's possible if an object has changed or renewed (https://blog.idrsolutions.com/multiple-trailers-in-a-pdf-file/), but in my case, the PDF files are not edited.
Does anyone know why all of the analysed files have two trailers?
This is a PDF file that contains a lot of text and also two images (there are two trailers in this file, who are (almost) identical to each other: :
0001a30bh: 74 72 61 69 6C 65 72 0D 0A 3C 3C 2F 53 69 7A 65 ; TRAILER..<</Size
0001a31bh: 20 34 37 2F 52 6F 6F 74 20 31 20 30 20 52 2F 49 ; 47/Root 1 0 R/I
0001a32bh: 6E 66 6F 20 31 35 20 30 20 52 2F 49 44 5B 3C 45 ; nfo 15 0 R/ID[<E
0001a33bh: 42 33 46 46 33 41 31 45 33 37 33 43 36 34 45 39 ; B3FF3A1E373C64E9
0001a34bh: 31 30 45 33 46 42 43 34 45 37 38 39 31 33 43 3E ; 10E3FBC4E78913C>
0001a35bh: 3C 45 42 33 46 46 33 41 31 45 33 37 33 43 36 34 ; <EB3FF3A1E373C64
0001a36bh: 45 39 31 30 45 33 46 42 43 34 45 37 38 39 31 33 ; E910E3FBC4E78913
0001a37bh: 43 3E 5D 20 3E 3E 0D 0A 73 74 61 72 74 78 72 65 ; C>] >>..startxre
0001a38bh: 66 0D 0A 31 30 36 33 32 33 0D 0A 25 25 45 4F 46 ; f..106323..%%EOF
0001a39bh: 0D 0A 78 72 65 66 0D 0A 30 20 30 0D 0A 74 72 61 ; ..xref..0 0..TRA
0001a3abh: 69 6C 65 72 0D 0A 3C 3C 2F 53 69 7A 65 20 34 37 ; ILER..<</Size 47
0001a3bbh: 2F 52 6F 6F 74 20 31 20 30 20 52 2F 49 6E 66 6F ; /Root 1 0 R/Info
0001a3cbh: 20 31 35 20 30 20 52 2F 49 44 5B 3C 45 42 33 46 ; 15 0 R/ID[<EB3F
0001a3dbh: 46 33 41 31 45 33 37 33 43 36 34 45 39 31 30 45 ; F3A1E373C64E910E
0001a3ebh: 33 46 42 43 34 45 37 38 39 31 33 43 3E 3C 45 42 ; 3FBC4E78913C><EB
0001a3fbh: 33 46 46 33 41 31 45 33 37 33 43 36 34 45 39 31 ; 3FF3A1E373C64E91
0001a40bh: 30 45 33 46 42 43 34 45 37 38 39 31 33 43 3E 5D ; 0E3FBC4E78913C>]
0001a41bh: 20 2F 50 72 65 76 20 31 30 36 33 32 33 2F 58 52 ; /Prev 106323/XR
0001a42bh: 65 66 53 74 6D 20 31 30 35 39 37 32 3E 3E 0D 0A ; efStm 105972>>..
0001a43bh: 73 74 61 72 74 78 72 65 66 0D 0A 31 30 37 34 32 ; startxref..10742
0001a44bh: 31 0D 0A 25 25 45 4F 46 ; 1..%%EOF
This is a PDF file that does only contain some random characters:
000071cbh: 74 72 61 69 6C 65 72 0D 0A 3C 3C 2F 53 69 7A 65 ; TRAILER..<</Size
000071dbh: 20 32 33 2F 52 6F 6F 74 20 31 20 30 20 52 2F 49 ; 23/Root 1 0 R/I
000071ebh: 6E 66 6F 20 39 20 30 20 52 2F 49 44 5B 3C 39 46 ; nfo 9 0 R/ID[<9F
000071fbh: 46 31 32 45 31 43 30 41 35 36 44 42 34 38 41 33 ; F12E1C0A56DB48A3
0000720bh: 41 31 43 37 32 30 33 38 32 33 30 32 45 32 3E 3C ; A1C720382302E2><
0000721bh: 39 46 46 31 32 45 31 43 30 41 35 36 44 42 34 38 ; 9FF12E1C0A56DB48
0000722bh: 41 33 41 31 43 37 32 30 33 38 32 33 30 32 45 32 ; A3A1C720382302E2
0000723bh: 3E 5D 20 3E 3E 0D 0A 73 74 61 72 74 78 72 65 66 ; >] >>..startxref
0000724bh: 0D 0A 32 38 36 35 39 0D 0A 25 25 45 4F 46 0D 0A ; ..28659..%%EOF..
0000725bh: 78 72 65 66 0D 0A 30 20 30 0D 0A 74 72 61 69 6C ; xref..0 0..TRAIL
0000726bh: 65 72 0D 0A 3C 3C 2F 53 69 7A 65 20 32 33 2F 52 ; ER..<</Size 23/R
0000727bh: 6F 6F 74 20 31 20 30 20 52 2F 49 6E 66 6F 20 39 ; oot 1 0 R/Info 9
0000728bh: 20 30 20 52 2F 49 44 5B 3C 39 46 46 31 32 45 31 ; 0 R/ID[<9FF12E1
0000729bh: 43 30 41 35 36 44 42 34 38 41 33 41 31 43 37 32 ; C0A56DB48A3A1C72
000072abh: 30 33 38 32 33 30 32 45 32 3E 3C 39 46 46 31 32 ; 0382302E2><9FF12
000072bbh: 45 31 43 30 41 35 36 44 42 34 38 41 33 41 31 43 ; E1C0A56DB48A3A1C
000072cbh: 37 32 30 33 38 32 33 30 32 45 32 3E 5D 20 2F 50 ; 720382302E2>] /P
000072dbh: 72 65 76 20 32 38 36 35 39 2F 58 52 65 66 53 74 ; rev 28659/XRefSt
000072ebh: 6D 20 32 38 33 37 34 3E 3E 0D 0A 73 74 61 72 74 ; m 28374>>..start
000072fbh: 78 72 65 66 0D 0A 32 39 32 37 35 0D 0A 25 25 45 ; xref..29275..%%E
0000730bh: 4F 46 ; OF
Those files are most likely created by MS Word. The excerpts you posted look like their interpretation of hybrid reference PDFs.
There are two special constructs in which the PDF specification uses the mechanisms it introduced for incremental updates for something else:
Linearized PDFs (see ISO 32000-2:2020 Annex F) and
Hybrid-reference PDFs (see ISO 32000-2:2020 Section 7.5.8.4).
Your excerpts look like the latter type of PDFs.
Some backgrounds:
With PDF 1.5 Adobe introduced the option to collect multiple non-stream indirect objects in a stream, a so called "object stream". The advantage of doing so is that data in streams can be compressed while otherwise those object cannot be compressed. At the same time they also introduced the option to put the cross reference table data into streams, the so called "cross-reference streams", also to allow compression.
Obviously a new type of cross reference entry was necessary to describe indirect objects in object streams, so they defined entries of that kind, but only for the cross-reference streams, not for the old cross reference tables.
PDFs stored using object and cross-reference streams often indeed are much smaller than the same PDFs stored as regular indirect objects with cross reference tables. On the other hand PDF processors that were not aware of these techniques couldn't open these PDFs at all.
Thus, Adobe came up with the idea of hybrid files: Files that contain the basic objects in a PDF required to view it at all in the old-fashioned way and the objects for newer or optional features in object and cross reference streams. The trailers of the cross reference tables contain an entry XRefStm pointing to the cross reference stream.
For some reason, though, it was specified that object lookup first had to be attempted in the cross reference table, and only if no entry was found there for the object number in question, the associated cross reference stream was to be searched.
As the first cross reference table is required to cover the complete range of object numbers used, this lookup strategy implied that hybrid-reference files needed a second cross reference table whose trailer could point to the cross reference stream that would be used for lookups before the innermost, first cross reference table.
This is what we see in your example:
trailer
<</Size 47/Root 1 0 R/Info 15 0 R/ID[<EB3FF3A1E373C64E910E3FBC4E78913C><EB3FF3A1E373C64E910E3FBC4E78913C>] >>
startxref
106323
%%EOF
xref
0 0
trailer
<</Size 47/Root 1 0 R/Info 15 0 R/ID[<EB3FF3A1E373C64E910E3FBC4E78913C><EB3FF3A1E373C64E910E3FBC4E78913C>] /Prev 106323/XRefStm 105972>>
startxref
107421
%%EOF
Actually most PDF producers implemented hybrid-reference files (if they did at all) under the impression that the cross reference stream and probably also the object streams should go between the first trailer and the second cross reference table. But there is no requirement for that, and the PDF export of MS Office chose to put all the streams before the first cross reference table. As that's the case for your examples, too, I assume they were produced by MS Office.
The first block of code below is an example of how I want my output to look like when I run my hexdump command in C. Atm, I don't know how to get the 3 columns of double spaces and I don't know how to add the ASCII representation and "." for non-ASCII of the bytes on the right.
hexdump -C /etc/passwd
00000000 72 6f 6f 74 3a 78 3a 30 3a 30 3a 72 6f 6f 74 3a |root:x:0:0:root:|
00000010 2f 72 6f 6f 74 3a 2f 62 69 6e 2f 62 61 73 68 0a |/root:/bin/bash.|
00000020 64 61 65 6d 6f 6e 3a 78 3a 31 3a 31 3a 64 61 65 |daemon:x:1:1:dae|
00000030 6d 6f 6e 3a 2f 75 73 72 2f 73 62 69 6e 3a 2f 62 |mon:/usr/sbin:/b|
00000040 69 6e 2f 73 68 0a 62 69 6e 3a 78 3a 32 3a 32 3a |in/sh.bin:x:2:2:|
00000050 62 69 6e 3a 2f 62 69 6e 3a 2f 62 69 6e 2f 73 68 |bin:/bin:/bin/sh|
This is my actual output:
00000000 2f 2a 20 68 65 6c 6c 6f 2e 63 20 2a 2f a. a. 23
00000010 69 6e 63 6c 75 64 65 20 3c 73 74 64 69 6f 2e 68
00000020 3e a. a. 69 6e 74 20 6d 61 69 6e 28 69 6e 74 20
00000030 61 72 67 63 2c 20 63 68 61 72 20 2a 61 72 67 76
00000040 5b 5d 29 a. 7b a. 20 20 70 75 74 73 28 22 48 65
00000050 6c 6c 6f 20 77 6f 72 6c 64 21 5c 6e 22 29 3b a.
00000060 20 20 72 65 74 75 72 6e 20 30 3b a. 7d a.
0000006e
thanks
I don't know how to get the 3 columns of double spaces
Just emit an extra space in the appropriate place.
The first one (between the offset and the first octet) is always there.
The second one is emitted before the 8th octet if there is one.
The third one is also always there.
You can either build them into a format string or bake them into your octet-formatting loop.
I don't know how to add the ASCII representation and "." for non-ASCII of the bytes on the right
Roughly, by using isprint() and then printing toprint instead of the raw character
char toprint = isprint(raw) ? raw : '.';
I am reading text from a file and outputting it as binary. I have modified the binary conversion as per follows:
Each capital letter shall start with 01 and will be followed by 5 bits.
The 5 bits shall hold the value of the letter.
The letters will have the value as A-2,B-3,C-4,D-5...
For example: HI-> (0101001)(0101010)
My code snippet is as follows:
void printinbits(int n)
{
for (int c = 4; c >= 0; c--)
{
long int k = n >> c;
if (k & 1)
printf("1");
else
printf("0");
}
}
int main()
{
//first letter is being repeated
char check[200];
FILE*fin= fopen("/Users/priya/Desktop/test.txt.rtf","r");
while((fscanf(fin,"%199s",check))==1)
{
for(int i=0;i<strlen(check);++i)
{
if(check[i]>=65&&check[i]<=90)
{
printf("01");
int n=check[i];
n-=63;
printinbits(n);
}
}
}
return 0;
}
My input->
HELLO
My output->
(0101001)(0101001)(0100110)(0101101)(0101101)(0110000)
(As you can see, the first letter H is being repeated)(Various letters are separated by brackets)
Here's a hex dump of a file hello.rtf containing the word HELLO in upper case. It was generated by TextEdit on a Mac.
0x0000: 7B 5C 72 74 66 31 5C 61 6E 73 69 5C 61 6E 73 69 {\rtf1\ansi\ansi
0x0010: 63 70 67 31 32 35 32 5C 63 6F 63 6F 61 72 74 66 cpg1252\cocoartf
0x0020: 31 34 30 34 5C 63 6F 63 6F 61 73 75 62 72 74 66 1404\cocoasubrtf
0x0030: 34 36 30 0A 7B 5C 66 6F 6E 74 74 62 6C 5C 66 30 460.{\fonttbl\f0
0x0040: 5C 66 73 77 69 73 73 5C 66 63 68 61 72 73 65 74 \fswiss\fcharset
0x0050: 30 20 48 65 6C 76 65 74 69 63 61 3B 7D 0A 7B 5C 0 Helvetica;}.{\
0x0060: 63 6F 6C 6F 72 74 62 6C 3B 5C 72 65 64 32 35 35 colortbl;\red255
0x0070: 5C 67 72 65 65 6E 32 35 35 5C 62 6C 75 65 32 35 \green255\blue25
0x0080: 35 3B 7D 0A 5C 6D 61 72 67 6C 31 34 34 30 5C 6D 5;}.\margl1440\m
0x0090: 61 72 67 72 31 34 34 30 5C 76 69 65 77 77 31 30 argr1440\vieww10
0x00A0: 38 30 30 5C 76 69 65 77 68 38 34 30 30 5C 76 69 800\viewh8400\vi
0x00B0: 65 77 6B 69 6E 64 30 0A 5C 70 61 72 64 5C 74 78 ewkind0.\pard\tx
0x00C0: 37 32 30 5C 74 78 31 34 34 30 5C 74 78 32 31 36 720\tx1440\tx216
0x00D0: 30 5C 74 78 32 38 38 30 5C 74 78 33 36 30 30 5C 0\tx2880\tx3600\
0x00E0: 74 78 34 33 32 30 5C 74 78 35 30 34 30 5C 74 78 tx4320\tx5040\tx
0x00F0: 35 37 36 30 5C 74 78 36 34 38 30 5C 74 78 37 32 5760\tx6480\tx72
0x0100: 30 30 5C 74 78 37 39 32 30 5C 74 78 38 36 34 30 00\tx7920\tx8640
0x0110: 5C 70 61 72 64 69 72 6E 61 74 75 72 61 6C 5C 70 \pardirnatural\p
0x0120: 61 72 74 69 67 68 74 65 6E 66 61 63 74 6F 72 30 artightenfactor0
0x0130: 0A 0A 5C 66 30 5C 66 73 32 34 20 5C 63 66 30 20 ..\f0\fs24 \cf0
0x0140: 48 45 4C 4C 4F 7D HELLO}
0x0146:
You may or may not be able to see the H of 'Helvetica' as the only other capital letter in the file — that would account for producing the output for HHELLO. It looks like you might be on a Mac too, so maybe you'd see the same result — or, at least, an equivalent one. (I used a homebrew hex dump program; you'd probably use xxd -g 1 test.txt.rtf, which would produce the hex with lower-case letters, and wouldn't include the final byte count line.)
You could, and should, print the data that your program reads in the loop, at least while debugging it, so that you can see what the program is processing. This is a very basic debugging technique.
In TextEdit, you can switch between rich text and plain text with the 'Make Plain Text' or 'Make Rich Text' option under the Format menu, or using ⇧⌘T (shift command T) to toggle between the two modes. Note how the file name changes as you do that.
Community Wiki since M Oehm pointed out the likely problem.
I'm new to C and system programming. I want to open an archive file and print out the name of the files inside the archive file (e.g., my archive file is weds.a; inside weds.a, I have thurs.txt and fri.txt". I want to create an output that shows
thurs.txt
fri.txt
EDITED: It should work like the ar -t command.
Can someone give me some tips on how to do it? I've been reading the man page and looking for examples online, but I'm getting no where. I believe I'm missing something. The code I have below only prints the link count. Can someone help? Thanks in advance for your help!!
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>
#include <errno.h>
#include <sys/utsname.h>
#include <ctype.h>
#include <string.h>
int main (int argc, char **argv)
{
int in_fd;
struct stat sb;
if (argc != 2) {
printf("Error", argv[0]);
exit(EXIT_FAILURE);
}
if (stat(argv[1], &sb) == -1) {
perror("stat");
exit(EXIT_FAILURE); //change from EXIT_SUCCESS to EXIT_FAILURE
}
//open the archive file (e.g., hw.a)
in_fd = open(argv[1], O_RDONLY);
if (in_fd == -1)
{
perror("Can't open input file\n");
exit(-1);
}
printf("Link Count: %ld\n", (long)sb.st_nlink);
return 0;
}
The easiest way is to use the ar program to list the names:
ar -tv weds.a
The - is optional; the v means you'll get size and time information.
If you want to get into reading the archive file itself, you'll have to be aware of the differences in the formats on different systems. The relevant header is (normally) <ar.h>. I have information for a number of systems, many of them obsolete, and there are a variety of different tricks used for handling long file names (and other even more basic file format issues) but you may have a more limited scope in mind. Any such work based on <ar.h> will be non-trivial; you're best off reusing what already exists (the ar program) if at all possible.
This is an archive from a Mac OS X 10.8.4 machine.
$ cat thurs.txt
0123456789:;<=>?#ABCDEFGHIJKLMNO
$ cat fri.txt
PQRSTUVWXYZ[\]^_`abcdefghijklmno
$ odx weds.a
0x0000: 21 3C 61 72 63 68 3E 0A 74 68 75 72 73 2E 74 78 !<arch>.thurs.tx
0x0010: 74 20 20 20 20 20 20 20 31 33 37 34 30 39 36 30 t 13740960
0x0020: 31 32 20 20 32 38 37 36 20 20 35 30 30 30 20 20 12 2876 5000
0x0030: 31 30 30 36 34 34 20 20 33 33 20 20 20 20 20 20 100644 33
0x0040: 20 20 60 0A 30 31 32 33 34 35 36 37 38 39 3A 3B `.0123456789:;
0x0050: 3C 3D 3E 3F 40 41 42 43 44 45 46 47 48 49 4A 4B <=>?#ABCDEFGHIJK
0x0060: 4C 4D 4E 4F 0A 0A 66 72 69 2E 74 78 74 20 20 20 LMNO..fri.txt
0x0070: 20 20 20 20 20 20 31 33 37 34 30 39 36 30 30 35 1374096005
0x0080: 20 20 32 38 37 36 20 20 35 30 30 30 20 20 31 30 2876 5000 10
0x0090: 30 36 34 34 20 20 33 33 20 20 20 20 20 20 20 20 0644 33
0x00A0: 60 0A 50 51 52 53 54 55 56 57 58 59 5A 5B 5C 5D `.PQRSTUVWXYZ[\]
0x00B0: 5E 5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D ^_`abcdefghijklm
0x00C0: 6E 6F 0A 0A no..
0x00C4:
$
Fortunately for you, the same files produce essentially the same archive on Linux too. In the Linux header <ar.h> you find:
/* Archive files start with the ARMAG identifying string. Then follows a
`struct ar_hdr', and as many bytes of member file data as its `ar_size'
member indicates, for each member file. */
#define ARMAG "!<arch>\n" /* String that begins an archive file. */
#define SARMAG 8 /* Size of that string. */
#define ARFMAG "`\n" /* String in ar_fmag at end of each header. */
struct ar_hdr
{
char ar_name[16]; /* Member file name, sometimes / terminated. */
char ar_date[12]; /* File date, decimal seconds since Epoch. */
char ar_uid[6], ar_gid[6]; /* User and group IDs, in ASCII decimal. */
char ar_mode[8]; /* File mode, in ASCII octal. */
char ar_size[10]; /* File size, in ASCII decimal. */
char ar_fmag[2]; /* Always contains ARFMAG. */
};
The Mac OS X header has the same structure and ARMAG and ARFMAG values, but one extra macro:
#define AR_EFMT1 "#1/" /* extended format #1 */
You can see the ARMAG string at the start of the file. Each file is then preceded by a struct ar_hdr. Note that the example names here are blank terminated, not slash terminated.
After that, you find the data for the file. You can read the header in its entirety. Note that if any of the names reaches above 15 characters, or if a name contains spaces, then you get an extra entry at the start of the archive file that contains the file name strings, and you also get a modified name entry in the per-file header that identifies the relevant string in the string table.
Linux archive with long names etc
0x0000: 21 3C 61 72 63 68 3E 0A 2F 2F 20 20 20 20 20 20 !<arch>.//
0x0010: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
* (1)
0x0030: 20 20 20 20 20 20 20 20 34 36 20 20 20 20 20 20 46
0x0040: 20 20 60 0A 66 69 6C 74 65 72 2E 73 74 64 65 72 `.filter.stder
0x0050: 72 2E 73 68 2F 0A 6C 6F 6E 67 20 6E 61 6D 65 20 r.sh/.long name
0x0060: 77 69 74 68 20 73 70 61 63 65 73 2E 74 78 74 2F with spaces.txt/
0x0070: 0A 0A 74 68 75 72 73 2E 74 78 74 2F 20 20 20 20 ..thurs.txt/
0x0080: 20 20 31 33 37 34 30 39 36 32 31 31 20 20 31 39 1374096211 19
0x0090: 39 34 38 34 35 30 30 30 20 20 31 30 30 36 34 30 94845000 100640
0x00A0: 20 20 33 33 20 20 20 20 20 20 20 20 60 0A 30 31 33 `.01
0x00B0: 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F 40 41 23456789:;<=>?#A
0x00C0: 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F 0A 0A BCDEFGHIJKLMNO..
0x00D0: 66 72 69 2E 74 78 74 2F 20 20 20 20 20 20 20 20 fri.txt/
0x00E0: 31 33 37 34 30 39 36 31 39 37 20 20 31 39 39 34 1374096197 1994
0x00F0: 38 34 35 30 30 30 20 20 31 30 30 36 34 30 20 20 845000 100640
0x0100: 33 33 20 20 20 20 20 20 20 20 60 0A 50 51 52 53 33 `.PQRS
0x0110: 54 55 56 57 58 59 5A 5B 5C 5D 5E 5F 60 61 62 63 TUVWXYZ[\]^_`abc
0x0120: 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F 0A 0A 2F 30 defghijklmno../0
0x0130: 20 20 20 20 20 20 20 20 20 20 20 20 20 20 31 33 13
0x0140: 37 31 31 34 35 35 38 34 20 20 31 39 39 34 38 34 71145584 199484
0x0150: 35 30 30 30 20 20 31 30 30 36 34 30 20 20 32 33 5000 100640 23
0x0160: 30 20 20 20 20 20 20 20 60 0A 23 21 2F 62 69 6E 0 `.#!/bin
0x0170: 2F 62 61 73 68 0A 73 65 74 20 2D 78 0A 72 6D 20 /bash.set -x.rm
0x0180: 2D 66 20 6F 75 74 2E 5B 31 32 33 5D 0A 2E 2F 67 -f out.[123]../g
0x0190: 65 6E 6F 75 74 65 72 72 2E 73 68 20 31 3E 2F 64 enouterr.sh 1>/d
0x01A0: 65 76 2F 6E 75 6C 6C 0A 2E 2F 67 65 6E 6F 75 74 ev/null../genout
0x01B0: 65 72 72 2E 73 68 20 32 3E 2F 64 65 76 2F 6E 75 err.sh 2>/dev/nu
0x01C0: 6C 6C 0A 28 20 2E 2F 67 65 6E 6F 75 74 65 72 72 ll.( ./genouterr
0x01D0: 2E 73 68 20 32 3E 26 31 20 31 3E 26 33 20 7C 20 .sh 2>&1 1>&3 |
0x01E0: 67 72 65 70 20 27 5B 30 2D 39 5D 30 27 20 3E 26 grep '[0-9]0' >&
0x01F0: 32 29 20 33 3E 6F 75 74 2E 33 20 32 3E 6F 75 74 2) 3>out.3 2>out
0x0200: 2E 32 20 31 3E 6F 75 74 2E 31 0A 6C 73 20 2D 6C .2 1>out.1.ls -l
0x0210: 20 6F 75 74 2E 5B 31 32 33 5D 0A 28 20 2E 2F 67 out.[123].( ./g
0x0220: 65 6E 6F 75 74 65 72 72 2E 73 68 20 32 3E 26 31 enouterr.sh 2>&1
0x0230: 20 31 3E 26 33 20 7C 20 67 72 65 70 20 27 5B 30 1>&3 | grep '[0
0x0240: 2D 39 5D 30 27 20 3E 26 32 29 20 33 3E 26 31 0A -9]0' >&2) 3>&1.
0x0250: 2F 31 38 20 20 20 20 20 20 20 20 20 20 20 20 20 /18
0x0260: 31 33 37 34 30 39 36 35 37 37 20 20 31 39 39 34 1374096577 1994
0x0270: 38 34 35 30 30 30 20 20 31 30 30 36 34 30 20 20 845000 100640
0x0280: 33 33 20 20 20 20 20 20 20 20 60 0A 30 31 32 33 33 `.0123
0x0290: 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F 40 41 42 43 456789:;<=>?#ABC
0x02A0: 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F 0A 0A DEFGHIJKLMNO..
0x02AE:
Mac OS X archive with long names etc
0x0000: 21 3C 61 72 63 68 3E 0A 74 68 75 72 73 2E 74 78 !<arch>.thurs.tx
0x0010: 74 20 20 20 20 20 20 20 31 33 37 34 30 39 36 30 t 13740960
0x0020: 31 32 20 20 32 38 37 36 20 20 35 30 30 30 20 20 12 2876 5000
0x0030: 31 30 30 36 34 34 20 20 33 33 20 20 20 20 20 20 100644 33
0x0040: 20 20 60 0A 30 31 32 33 34 35 36 37 38 39 3A 3B `.0123456789:;
0x0050: 3C 3D 3E 3F 40 41 42 43 44 45 46 47 48 49 4A 4B <=>?#ABCDEFGHIJK
0x0060: 4C 4D 4E 4F 0A 0A 66 72 69 2E 74 78 74 20 20 20 LMNO..fri.txt
0x0070: 20 20 20 20 20 20 31 33 37 34 30 39 36 30 30 35 1374096005
0x0080: 20 20 32 38 37 36 20 20 35 30 30 30 20 20 31 30 2876 5000 10
0x0090: 30 36 34 34 20 20 33 33 20 20 20 20 20 20 20 20 0644 33
0x00A0: 60 0A 50 51 52 53 54 55 56 57 58 59 5A 5B 5C 5D `.PQRSTUVWXYZ[\]
0x00B0: 5E 5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D ^_`abcdefghijklm
0x00C0: 6E 6F 0A 0A 66 69 6C 74 65 72 2E 73 74 64 65 72 no..filter.stder
0x00D0: 72 2E 73 68 31 33 37 34 30 39 37 37 39 34 20 20 r.sh1374097794
0x00E0: 32 38 37 36 20 20 35 30 30 30 20 20 31 30 30 36 2876 5000 1006
0x00F0: 34 34 20 20 32 33 30 20 20 20 20 20 20 20 60 0A 44 230 `.
0x0100: 23 21 2F 62 69 6E 2F 62 61 73 68 0A 73 65 74 20 #!/bin/bash.set
0x0110: 2D 78 0A 72 6D 20 2D 66 20 6F 75 74 2E 5B 31 32 -x.rm -f out.[12
0x0120: 33 5D 0A 2E 2F 67 65 6E 6F 75 74 65 72 72 2E 73 3]../genouterr.s
0x0130: 68 20 31 3E 2F 64 65 76 2F 6E 75 6C 6C 0A 2E 2F h 1>/dev/null../
0x0140: 67 65 6E 6F 75 74 65 72 72 2E 73 68 20 32 3E 2F genouterr.sh 2>/
0x0150: 64 65 76 2F 6E 75 6C 6C 0A 28 20 2E 2F 67 65 6E dev/null.( ./gen
0x0160: 6F 75 74 65 72 72 2E 73 68 20 32 3E 26 31 20 31 outerr.sh 2>&1 1
0x0170: 3E 26 33 20 7C 20 67 72 65 70 20 27 5B 30 2D 39 >&3 | grep '[0-9
0x0180: 5D 30 27 20 3E 26 32 29 20 33 3E 6F 75 74 2E 33 ]0' >&2) 3>out.3
0x0190: 20 32 3E 6F 75 74 2E 32 20 31 3E 6F 75 74 2E 31 2>out.2 1>out.1
0x01A0: 0A 6C 73 20 2D 6C 20 6F 75 74 2E 5B 31 32 33 5D .ls -l out.[123]
0x01B0: 0A 28 20 2E 2F 67 65 6E 6F 75 74 65 72 72 2E 73 .( ./genouterr.s
0x01C0: 68 20 32 3E 26 31 20 31 3E 26 33 20 7C 20 67 72 h 2>&1 1>&3 | gr
0x01D0: 65 70 20 27 5B 30 2D 39 5D 30 27 20 3E 26 32 29 ep '[0-9]0' >&2)
0x01E0: 20 33 3E 26 31 0A 23 31 2F 32 38 20 20 20 20 20 3>&1.#1/28
0x01F0: 20 20 20 20 20 20 31 33 37 34 30 39 37 38 32 32 1374097822
0x0200: 20 20 32 38 37 36 20 20 35 30 30 30 20 20 31 30 2876 5000 10
0x0210: 30 36 34 34 20 20 36 31 20 20 20 20 20 20 20 20 0644 61
0x0220: 60 0A 6C 6F 6E 67 20 6E 61 6D 65 20 77 69 74 68 `.long name with
0x0230: 20 73 70 61 63 65 73 2E 74 78 74 00 00 00 30 31 spaces.txt...01
0x0240: 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F 40 41 23456789:;<=>?#A
0x0250: 42 43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F 0A 0A BCDEFGHIJKLMNO..
0x0260:
Differences
The Linux archive has a list of strings at the top of the file that have to be remembered. The Mac OS X archive has the special entry #1/28 which identifies the header as being followed by a 28-byte entry containing the file name (null padded to a multiple of 4 bytes; the length given includes the null padding). The Mac archive has no space after the name when it is exactly 16 characters long; the Linux archive puts the 16-character name into the string table.
Im tring to export file from wireshark , so i could search in it.
now every option i try doesn't give simple raw format as the tcp raw view , when i follow
tcp stream .
all it gives me is hex view of the packets and the string in this kind of format breaks and can't be searchable . i want it to export to searchable format.
can it be done ?
this is what im getting now :
0000 48 54 54 50 2f 31 2e 31 20 35 30 30 20 49 6e 74 HTTP/1.1 500 Int
0010 65 72 6e 61 6c 20 53 65 72 76 65 72 20 45 72 72 ernal Server Err
0020 6f 72 0d 0a 44 61 74 65 3a 20 54 68 75 2c 20 31 or..Date: Thu, 1
0030 30 20 4e 6f 76 20 32 30 31 31 20 31 36 3a 33 32 0 Nov 2011 16:32
0040 3a 35 37 20 47 4d 54 0d 0a 50 72 61 67 6d 61 3a :57 GMT..Pragma:
0050 20 6e 6f 2d 63 61 63 68 65 0d 0a 43 6f 6e 74 65 no-cache..Conte
What about using TShark, sed and tr?
tshark -r Clmt_04.pcap -x -R "frame.number<40" | sed 's/^.{56}//' | tr -d '\n' > Clmt-04.txt
tshark -x
add output of hex and ASCII dump (Packet Bytes)
sed 's/^.{56}//'
remove the first 56 characters of each line
tr -d '\n'
remove new line character
Hope this helps
After you identify the tcp stream, you can use the following command with tshark:
tshark -nr <file>.pcapng -q -z follow,tcp,ascii,XXXX
Where XXXX is the tcp stream.