I am trying to import about 90000 json files into MongoDB using mongoimport. To import I use this command in powershell: cat *.json | mongoimport --db='myDB' --collection='myCollection'
But in the output, instead of some characters, question marks appear.
MongoDB doc
I have changed the encoding of my powershell and it outputs the contents of files with turkish letters well. Tell me how to fix this?
PowerShell Output
Related
I have a batch file (called single-line.bat) with the following contents
python -c "import math; print(math.hypot(3, 4))"
I want the argument to python to be a multiline string that is constructed across multiple lines of the batch file. That is, I'd like the contents to resemble this
python -c "import math
print(
math.hypot(
3,
4)
)"
That is, I want to build a string literal that contains multiple lines. I also want to build that string literal across multiple lines.
This answer builds a command across multiple lines, not a string literal: https://superuser.com/a/1025263
This answer builds a string across multiple lines, but the string itself does not contain multiple lines: https://superuser.com/a/1596233
This answer comes close, but since it does not use quotes, it looks like I to escape batch characters such as %.: Passing around multi-line strings
To be sure, the following is a valid python script:
import math
print(
math.hypot(
3,
4)
)
and that is the script that I want to pass as an argument to the -c flag of the python command in my batch file.
[EDIT]
An answer suggests making a file proposal-1.bat
python -c "import math"^
"print("^
"math.hypot("^
"3,"^
"4)"^
")"
This doesn't work. Compared to the original single-line.bat:
> single-line.bat
> python -c "import math; print(math.hypot(3, 4))"
5.0
> proposal-1.bat
> python -c "import math" "print(" "math.hypot(" "3," "4)" ")"
I do not see any output python.
I cannot successfully post this as a comment with the intended formatting, so here it is as a potential answer instead.
Have you tried:
python -c "import math;"^
"print("^
"math.hypot("^
"3,"^
"4)"^
")"
Please note that the leading spaces on the continuation lines are intentional.
I have an Azure SQL Server database and a linux box. I have a csv file on the linux machine that I want to import into SQL Server. I have a table already created where I am going to import this file. I have the following questions -
1) Why does this command return an Unknown argument: -S
bcp table in ~/test.csv -S databaseServerName -d dbName -U myUsername -q -c -t
2) How do I import only part of the csv file? It has 20 columns, but I only want to import 2.
3) My table has these two columns - State, Province. My csv file has these two columns that I want to import - State, Region. How do I get Province to map to region.
For #2 and #3, you need to use a BCP format file. This allows you column-level control over which fields from the file go to which columns in the destination and which are left behind (not given a destination).
Use the -f option of BCP and specify the location and name of the format file you want to use. Sorry, no help yet with #1. Have a few questions/suggestions. But im not that familiar with Linux environments.
For part 2 of your question, you can use the Linux cut command to extract just the columns you want. A short awk script can do the same thing (see this SO answer). For both of these, you'll have to identify the "State" and "Region" columns by number. A non-native solution is [querycsv.py][1], which can also rename the "Region" column (disclaimer: I wrote querycsv.py).
For part 3 of your question, you can use the Linux sed command to change the column name on the first line of the CSV file, e.g., sed -e "1s/Region/Province/" file.csv >file2.csv.
The US Naval Observatory has an API that outputs a JSON file containing the sunrise and sunset times, among other things, as documented here.
Here is an example of the output JSON file:
{
"error":false,
"apiversion":"2.0.0",
"year":2017,
"month":6,
"day":10,
"dayofweek":"Saturday",
"datechanged":false,
"lon":130.000000,
"lat":30.000000,
"tz":0,
"sundata":[
{"phen":"U", "time":"03:19"},
{"phen":"S", "time":"10:21"},
{"phen":"EC", "time":"10:48"},
{"phen":"BC", "time":"19:51"},
{"phen":"R", "time":"20:18"}],
"moondata":[
{"phen":"R", "time":"10:49"},
{"phen":"U", "time":"16:13"},
{"phen":"S", "time":"21:36"}],
"prevsundata":[
{"phen":"BC","time":"19:51"},
{"phen":"R","time":"20:18"}],
"closestphase":{"phase":"Full Moon","date":"June 9, 2017","time":"13:09"},
"fracillum":"99%",
"curphase":"Waning Gibbous"
}
I'm relatively new to using JSON, but I understand that everything in square brackets after "sundata" is a JSON array (please correct me if I'm wrong). So I searched for instructions on how to get a value from a JSON array, without success.
I have downloaded the file to my system using:
wget -O usno.json "http://api.usno.navy.mil/rstt/oneday?ID=iOnTheSk&date=today&tz=0&coords=30,130"
I need to extract the time (in HH:MM format) from this line:
{"phen":"S", "time":"10:21"},
...and then use it to create a variable (that I will later write to a separate file).
I would prefer to use Bash if possible, preferably using a JSON parser (such as jq) if it'll be easier to understand/implement. I'd rather not use Python (which was suggested by a lot of the articles I have read previously) if possible as I am trying to become more familiar with Bash specifically.
I have examined a lot of different webpages, including answers on Stack Overflow, but none of them have specifically covered an array line with two key/value pairs per line (they've only explained how to do it with only one pair per line, which isn't what the above file structure has, sadly).
Specifically, I have read these articles, but they did not solve my particular problem:
https://unix.stackexchange.com/questions/177843/parse-one-field-from-an-json-array-into-bash-array
Parsing JSON with Unix tools
Parse json array in shell script
Parse JSON to array in a shell script
What is JSON and why would I use it?
https://developers.squarespace.com/what-is-json/
Read the json data in shell script
Thanks in advance for any thoughts.
Side note: I have managed to do this with a complex 150-odd line script made up of "sed"s, "grep"s, "awk"s, and whatnot, but obviously if there's a one-liner JSON-native solution that's more elegant, I'd prefer to use that as I need to minimise power usage wherever possible (it's being run on a battery-powered device).
(Side-note to the side-note: the script was so long because I need to do it for each line in the JSON file, not just the "S" value)
If you already have jq you can easily select your desired time with:
sun_time=$(jq '.sundata[] | select(.phen == "S").time' usno.json)
echo $sun_time
# "10:21"
If you must use "regular" bash commands (really, use jq):
wget -O - "http://api.usno.navy.mil/rstt/oneday?ID=iOnTheSk&date=today&tz=0&coords=30,130" \
| sed -n '/^"sundata":/,/}],$/p' \
| sed -n -e '/"phen":"S"/{s/^.*"time":"//'\;s/...$//\;p}
Example:
$ wget -O - "http://api.usno.navy.mil/rstt/oneday?ID=iOnTheSk&date=today&tz=0&coords=30,130" | sed -n '/^"sundata":/,/}],$/p' | sed -n -e '/"phen":"S"/{s/^.*"time":"//'\;s/...$//\;p}
--2017-06-10 08:02:46-- http://api.usno.navy.mil/rstt/oneday?ID=iOnTheSk&date=today&tz=0&coords=30,130
Resolving api.usno.navy.mil (api.usno.navy.mil)... 199.211.133.93
Connecting to api.usno.navy.mil (api.usno.navy.mil)|199.211.133.93|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [application/json]
Saving to: ‘STDOUT’
- [ <=> ] 753 --.-KB/s in 0s
2017-06-10 08:02:47 (42.6 MB/s) - written to stdout [753]
10:21
I used r.tile (http://grass.osgeo.org/grass71/manuals/r.tile.html) to create tiles from a huge geotiff. Now i want to export each tile to a file but havent found an easy way:
Using r.out.gdal I only can export one tile at a time which makes it unuseable as I have 100 tiles...
I havent found any other solution...anyone any ideas?
You just call r.out.gdal in a for loop in some scripting language.
Here is a example for Bash. It is a simple but complete script. It expects the raster map names as command line parameters and if the parameters are not present it fails with a usage suggestion.
#!/usr/bin/env bash
if [ $# -eq 0 ]
then
>&2 echo "No arguments supplied"
>&2 echo "Usage: $0 raster1 raster2 ..."
fi
for RASTER in "$#"
do
r.out.gdal input=$RASTER output=$RASTER.tiff format=GTiff
done
To run it you have to set it as executable (chmod u+x export.sh) first, then you can run it in the command line:
./export.sh tile1 tile2
Here is an example for Python which is much more convented for bigger tasks. The code does completely the same as the Bash script. Additionally, it uses GRASS Python Scripting Library to call the r.in.gdal module and print function to be forward compatible with Python 3.
#!/usr/bin/env python
from __future__ import print_function
import sys
import grass.script as gscript
if len(sys.argv) == 1:
print("No arguments supplied", file=sys.stderr)
print("Usage: {} raster1 raster2 ...".format(sys.argv[0]), file=sys.stderr)
for raster in sys.argv[1:]:
gscript.run_command('r.out.gdal', input=raster,
output=raster + '.tiff', format='GTiff')
Again, to run it you have to set it as executable (chmod u+x export.py) first, then you can run it in the command line:
./export.py tile1 tile2
Both examples assume you are using Linux, Mac OS X or something similar and running the script in system command line in GRASS GIS. On MS Windows, you should probably use the Python version and run it, for example from the GRASS GIS GUI through the Launch script item in the File menu in the Layer Manager.
The above answers require you to input the name of each raster layer. If you want to automate the whole process you can use the following Python script. This script first creates a list of all rasters that are available to grass, it then exports each as a Tiff file individually. To run it, either copy and paste the code into the GRASS python shell (accessible as a tab on the GRASS GUI layer manager window) or, save it as a .py file and, again from the Python shell, use the command execfile("foo.py").
import grass.script as grass
# create list of all rasters
rastlist=grass.read_command("g.list",type="rast")
rastlist1= rastlist.split(':', 1)[1]
rastlist2= rastlist1.split('-', 1)[0]
RastListClean=rastlist2.split()
# export all rasters
for raster in RastListClean:
grass.run_command('r.out.gdal', input=raster,
output=raster + '.tiff', format='GTiff')
Working with GRASS 8.2 on macOS, this version works.
The g.list() seems to return a string, separated perhaps by newlines.
split() breaks it into a python list that can be iterated.
Note that this only works because GRASS does not allow spaces in object names.
#!/usr/bin/env python3
import grass.script as grass
rastlist=grass.read_command("g.list", type="rast")
rastlistClean=rastlist.split()
for raster in rastlistClean:
grass.run_command('r.out.gdal', input=raster, output= raster + '.tif', format='GTiff', overwrite=True)
Credit to Beeman's answer which is the basis for this one.
I just got knocked down after our server has been updated from Debian 4 to 5.
We switched to UTF-8 environment and now we have problems getting the text printed correctly on the browser, because all files are in non-utf8 encodings like iso-8859-1, ascii, etc.
I tried many different scripts.
The first one I tried is "iconv". That one doesn't work, it changes the content, but the file's encoding is still non-utf8.
Same problem with enca, encamv, convmv and some other tools I installed via apt-get.
Then I found a python code, which uses chardet Universal Detector module, to detect encoding of a file (which works fine), but using the unicode class or the codec class to save it as utf-8 doesn't work, without any errors.
The only way I found to get the file and its content converted to UTF-8, is vi.
These are the steps I do for one file:
vi filename.php
:set bomb
:set fileencoding=utf-8
:wq
That's it. That one works perfect. But how can I get this running via a script?
I would like to write a script (Linux shell) which traverses a directory taking all php files, then converting them using vi with the commands above.
As I need to start the vi app, I do not know how to do something like this:
"vi --run-command=':set bomb, :set fileencoding=utf-8' filename.php"
Hope someone can help me.
This is the simplest way I know of to do this easily from the command line:
vim +"argdo se bomb | se fileencoding=utf-8 | w" $(find . -type f -name *.php)
Or better yet if the number of files is expected to be pretty large:
find . -type f -name *.php | xargs vim +"argdo se bomb | se fileencoding=utf-8 | w"
You could put your commands in a file, let's call it script.vim:
set bomb
set fileencoding=utf-8
wq
Then you invoke Vim with the -S (source) option to execute the script on the file you wish to fix. To do this on a bunch of files you could do
find . -type f -name "*.php" -exec vim -S script.vim {} \;
You could also put the Vim commands on the command line using the + option, but I think it may be more readable like this.
Note: I have not tested this.
You may actually want set nobomb (BOM = byte order mark), especially in the [not windows] world.
e.g., I had a script that didn't work as there was a byte order mark at the start. It isn't usually displayed in editors (even with set list in vi), or on the console, so its difficult to spot.
The file looked like this
#!/usr/bin/perl
...
But trying to run it, I get
./filename
./filename: line 1: #!/usr/bin/perl: No such file or directory
Not displayed, but at the start of the file, is the 3 byte BOM. So, as far as linux is concerned, the file doesn't start with #!
The solution is
vi filename
:set nobomb
:set fileencoding=utf-8
:wq
This removes the BOM at the start of the file, making it correct utf8.
NB Windows uses the BOM to identify a text file as being utf8, rather than ANSI. Linux (and the official spec) doesn't.
The accepted answer will keep the last file open in Vim. This problem can be easily resolved using the -c option of Vim,
vim +"argdo set bomb | set fileencoding=utf-8 | w" -c ":q" file1.txt file2.txt
If you need only process one file, the following will also work,
vim -c ':set bomb' -c ':set fileencoding=utf-8' -c ':wq' file1.txt