My data is stored in Elasticsearch and I need a way to pull it and use SPSS with.
Is there any method for this? Maybe a connector layer of programming I need to do on SPSS?
Thanks in adv
The easiest way I see is to use Logstash to export your data from ES into a CSV file and then import that CSV into SPSS.
A sample Logstash config file to export your data would look like this:
input {
elasticsearch {
hosts => ["localhost:9200"]
query => '{"query":{"match_all":{}}'
index => "my-index"
scroll => "5m"
}
}
filter {
mutate {
remove_field => [ "#version", "#timestamp" ]
}
}
output {
csv {
fields => ["header1", "header2", "header3"]
path => "my-index.csv"
}
}
Related
I want the table name to be today's date.
For example: dbtest_2022.dbo.tbshop2_2022_08_22 auto change to dbtest_2022.dbo.tbshop2_2022_08_23
input {
jdbc {
jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/sqljdbc42.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "..."
jdbc_user => "..."
jdbc_password => "..."
statement => "SELECT ... FROM dbtest_2022.dbo.tbshop2_2022_08_22"
schedule => "*/14 * * * *"
add_field => { "tag" => "mssql-test" }
type => "mssql"
}
}
I had an idea to make two requests.
In the first request, get the table id and in the second, make a request with this id.
statement => "SELECT OBJECT_ID(N'dbtest_2022.dbo.tbshop2_{year}_{month}_{day}') AS 'object_id' AND SELECT ... FROM table(object_id)"
or is there any expression in jdbc to implement this?
For example: dbtest_2022.dbo.tbshop2_now()
or can I use dbtest_2022.dbo.tbshop2_{year}{month}{day} in jdbc ?
Well, so that every day you don’t change the request with your hands, but somehow automate it
I am not very good at SQL. I would be glad for help or ideas on how to implement this.
I solutioned with below steps.
1. [.profile] set Envoronment variable.
...
CURRENT_YYYYMM=`date "+%Y%m`
export CURRENT_YYYYMM
...
2. logstash config file
input {
jdbc {
...
FROM ${CURRENT_YYYYMM}_logs
...
}
reference :
https://www.elastic.co/guide/en/logstash/current/environment-variables.html
I've been trying to save data from my form in my MongoDB for some time.
I also get a response from the database.
See also: create object in mongo db api onclick sending form
Unfortunately there are not enough tutorials in my mother tongue and I don't seem to understand everything in English.
I've tried some of the documentation, but I always fail.
What is missing in my webhook function so that the form data can be stored?
exports = function(payload) {
const mongodb = context.services.get("mongodb-atlas");
const mycollection = mongodb.db("created_notifications").collection("dpvn_collection");
return mycollection.find({}).limit(10).toArray();
};
The Webhookfunction was totally wrong.
READ THE DOCUMENTATION FIRST
exports = function(payload, response) {
const mongodb = context.services.get("mongodb-atlas");
const requestLogs = mongodb.db("created_notifications").collection("dpvn_collection");
requestLogs.insertOne({
body: EJSON.parse(payload.body.text()),
query: payload.query
}).then(result => {
})
};
I am using react-table v7, useExportData and papaparse to download a .csv from a table referencing to this codesandbox, so far it downloaded successfully but I can't find a way to name a file to download. My code is following
function getExportFileBlob({ columns, data, fileType, fileName }) {
if (fileType === "csv") {
// CSV example
const headerNames = columns.map((col) => col.exportValue);
const csvString = Papa.unparse({ fields: headerNames, data });
return new Blob([csvString], { type: "text/csv" });
}
You can use the getExportFileName function defined in the docs and overwrite the default all-data.fileType or data.fileType name.
by returning a name for your exported data. Check out the following sandbox!
Hope this helps.
`
Probably too late, but for those seeking for an answer, according to docs,
getExportFileName: Function({ fileType, all }) => string
is the function you need to override the default file name.
I believe you can access it from main useTable where you are passing this plugin:
const {
....
exportData,
} = useTable(
{
getExportFileName: ({ all }) => {
return `${all ? 'exported-all' : 'current-view-only'}` // <--- there
},
},
useExportData
)
check it out, hope this helps!
I've just started using Gatsby with the Sanity headless CMS.
For the most part it's pretty straight forward; but knowing best practises for querying the data through GraphQL is still bothering me. How I'm doing it currently is just frantically clicking through my CMS structure in the GraphQL playground and finding what I want. This works but the lack of uniformity in this approach is making me uneasy.
For example, if I want a hero image that's in the CMS somewhere, i'll need to do something like:
query SomePageQuery($id: String) {
sanitySomePage(id: { eq: $id }) {
id
heroImage {
asset {
fluid(maxWidth: 1500) {
...GatsbySanityImageFluid
}
}
}
}
}
But if I want some PortableText block then I need to query the corresponding _raw field of whatever type. So, if my type was introText, Gatsby also provides a _rawIntroText. I'm only able to get the full PortableText from this _raw version of the data. Like this:
query SomePageQuery($id: String) {
sanitySomePage(id: { eq: $id }) {
id
_rawIntroText
}
}
It seems that, for some data you can use [Type], and sometimes you have to use _raw[Type].
There's not a great deal of documentation as to why this is the case. And I'm not sure if this is enforced via Sanity or Gatsby.
My question I guess would be, why does _raw[Anything] exist in the Gatsby and/or Sanity world, and how do people decide on which to use (other than just trial and error within the GraphQL playground and at runtime)?
This is coming from the gatsby-source-sanity plugin that Sanity built and maintains. Hopefully someone from Sanity can provide more context, but effectively the _raw[FieldName] entries return the original JSON data for the field. The unprefixed field (e.g. fieldName) is probably not what you want—it'll only contain bits of metadata about the data.
I tend to pull the _raw[FieldName] data and then just pass it straight into the #sanity/block-content-to-react component like so:
import React from "react"
import { graphql } from "gatsby"
import SanityBlockContent from "#sanity/block-content-to-react"
export default ({ data: { page } }) => (
<SanityBlockContent
blocks={page.textContent}
projectId={process.env.GATSBY_SANITY_PROJECT_ID}
dataset={process.env.GATSBY_SANITY_DATASET}
/>
)
export const query = graphql`
query SomePageQuery($id: String) {
page: sanitySomePage(id: { eq: $id }) {
textContent: _rawTextContent
}
}
`
Note that I'm using GraphQL aliasing to continue to refer to the field as textContent in my component rather than coupling the component to the specifics of this GraphQL schema.
You don't need to use Gatsby Image for Sanity images since they have their own image transformation pipeline anyways. Instead you can just fetch asset { _id } and then use #sanity/client like this to generate an image url:
import sanityClient from "#sanity/client"
import sanityImageUrl from "#sanity/image-url"
const client = sanityClient({
dataset: process.env.GATSBY_SANITY_DATASET,
projectId: process.env.GATSBY_SANITY_PROJECT_ID,
useCdn: true,
})
const builder = sanityImageUrl(client)
builder.image({ _id: "..." }).width(400).dpr(2).url()
could please someone explain to me, why logstash keeps ignoring "codec => plain => format" setting, I am trying to set?
Cfg file I am using:
input {
gelf {
host => "[some ip]"
port => 12201
}
}
output {
elasticsearch {
host => "[some ip]"
bind_port => "9301"
}
file {
codec => plain {
format => "%{[time]} | %{[severity]} : /%{[thread]}/ %{[loggername]} (%{[sourcemethodname]}) - %{[message]}"
}
path => "/Users/[some user]/logs/%{host}/%{facility}-%{+YYYY-MM-dd}.log"
}
}
I thought I used the wrong format, tried different combinations like "%{time}" for fields and even tried to use constant text like:
codec => plain {format => "Simple line"}
But nothing seems to work. It outputs to the elasticsearch fine, create folder/files, but outputs it as JSON.
If anyone knows what is going on with it, please help.
Thanks.
Parameter message_format is deprecated and will be remove in future relases of Logstash. Instead of using message_format try something like this:
file {
codec => line {
format => "%{[time]} | %{[severity]} : /%{[thread]}/ %{[loggername]} (%{[sourcemethodname]}) - %{[message]}"
}
path => "/Users/[some user]/logs/%{host}/%{facility}-%{+YYYY-MM-dd}.log"
}
PS: your example using codec plain, try my with line.
file has a message_format parameter that is what you'll want to use:
file {
message_format => "%{[time]} | %{[severity]} : /%{[thread]}/ %{[loggername]} (%{[sourcemethodname]}) - %{[message]}"
path => "/Users/[some user]/logs/%{host}/%{facility}-%{+YYYY-MM-dd}.log"
}