I want the table name to be today's date.
For example: dbtest_2022.dbo.tbshop2_2022_08_22 auto change to dbtest_2022.dbo.tbshop2_2022_08_23
input {
jdbc {
jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/sqljdbc42.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "..."
jdbc_user => "..."
jdbc_password => "..."
statement => "SELECT ... FROM dbtest_2022.dbo.tbshop2_2022_08_22"
schedule => "*/14 * * * *"
add_field => { "tag" => "mssql-test" }
type => "mssql"
}
}
I had an idea to make two requests.
In the first request, get the table id and in the second, make a request with this id.
statement => "SELECT OBJECT_ID(N'dbtest_2022.dbo.tbshop2_{year}_{month}_{day}') AS 'object_id' AND SELECT ... FROM table(object_id)"
or is there any expression in jdbc to implement this?
For example: dbtest_2022.dbo.tbshop2_now()
or can I use dbtest_2022.dbo.tbshop2_{year}{month}{day} in jdbc ?
Well, so that every day you don’t change the request with your hands, but somehow automate it
I am not very good at SQL. I would be glad for help or ideas on how to implement this.
I solutioned with below steps.
1. [.profile] set Envoronment variable.
...
CURRENT_YYYYMM=`date "+%Y%m`
export CURRENT_YYYYMM
...
2. logstash config file
input {
jdbc {
...
FROM ${CURRENT_YYYYMM}_logs
...
}
reference :
https://www.elastic.co/guide/en/logstash/current/environment-variables.html
Related
I'm having some trouble trying to achieve multiple connection to database in some clean way.
Keep in mind that this is my first symfony project ever, and i'm only a young developer.
In my project, the goal is to be able to select a client, with a specific database, and to connect to the database to be able to export some datas.
I tried to do the solution describe in this post Symfony 3 connection to multiple databases and i tried to generate dynamically an entityManager.
So i created a factory EntityManagerFactory :
Factory\EntityManagerFactory
<?php
namespace App\Factory;
use Doctrine\ORM\Tools\Setup;
use Doctrine\ORM\EntityManager;
use Symfony\Component\Yaml\Yaml;
class EntityManagerFactory {
private $config_db_group;
public function __construct(string $config_db_group) {
$this->config_db_group = $config_db_group;
}
public function createManager($idDb) {
$isDevMode = false;
$config = Setup::createAnnotationMetadataConfiguration(array(__DIR__ . "/src"), $isDevMode);
$connectionConfig = $this->getConfigDb($idDb);
$dbParams = [
'driver' => 'pdo_mysql',
'host' => $connectionConfig['host'],
'username' => $connectionConfig['user'],
'password' => $connectionConfig['password'],
'dbname' => $connectionConfig['db_name']
];
return EntityManager::create($dbParams, $config);
}
private function getConfigDb($idDb) {
$connectionConfig = Yaml::parseFile("$this->config_db_group");
return $connectionConfig[$idDb];
}
}
I have a yaml that describes the connection config :
config\dbgroup.yaml
1:
db_name: "db_name1"
host: "host1"
user: "user1"
password: "password1"
port: "3306"
2:
db_name: "db_name2"
host: "host2"
user: "user2"
password: "password2"
port: "3306"
In my config\services.yaml, i did something that was described in the post.
# Create a service for the factory
App\Factory\EntityManagerFactory:
arguments:
$config_db_group: '%kernel.project_dir%\config\db_group.yaml'
# Use the factory service as the first argument of the 'factory' option
# and the factory method as the second argument
App\Factory\EntityManager:
factory: ['#App\Factory\EntityManagerFactory', 'getManager']
I don't really understand what this is, i think this defines my factory as a service ? ...
And then i try to create an entityManager in my controller, this was just to test if it works, i think database request should be in a Repository, or a Services ?
<?php
namespace App\Controller;
use Twig\Environment;
use App\Factory\EntityManagerFactory;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\Routing\Annotation\Route;
use App\Repository\Istrator\DatabaseGroupRepository;
use Doctrine\ORM\EntityManager;
use Doctrine\ORM\Query\ResultSetMapping;
use Doctrine\ORM\Query\ResultSetMappingBuilder;
use Symfony\Bundle\FrameworkBundle\Controller\AbstractController;
class DashboardController extends AbstractController {
private $twig;
private $databaseGroupRepository;
private $factory;
public function __construct(Environment $twig, DatabaseGroupRepository $databaseGroupRepository, EntityManagerFactory $factory) {
$this->twig = $twig;
$this->databaseGroupRepository = $databaseGroupRepository;
$this->factory = $factory;
}
#[Route('{slug}/dashboard', name: 'app_dashboard')]
public function index(string $slug): Response {
// I get the specific database
$databaseGroup = $this->databaseGroupRepository->findBySlug($slug);
// Then i try to create an entityManager with the correct config
$entityManager = $this->factory->createManager($databaseGroup->getIdDb());
// Then just to try my connection, i create a basic query
$rsm = new ResultSetMapping();
$test = $entityManager->createNativeQuery("USE mydatabase; SELECT * FROM mytable", $rsm)->execute();
return new Response($this->twig->render('dashboard/dashboard.html.twig', [
'controller_name' => 'DashboardController',
]));
}
}
For now it doesn't work.. But i have several questions :
The databases i try to connect are not database with the same database schemes than my actual database. They are external database. Should i create Entities, and repository to manage them ? Or should i just do some connection, some request without entity and repository ?
In the stackoverflow post that i based my code on, there is a second way of doing it, by defining all future connection in a doctrine.yaml. I have a defined number of connection but like 50 or something, should i do this instead of creating entityManager dynamically ?
As you can see, i'm a bit confused right now but if someone could tell me their point of vue, it would be great.
If you need any other information, just tell me !
Thanks in advance
EDIT :
I found the solution, and it was really stupid :
In my EntityManagerFactory, i did this :
$dbParams = [
'driver' => 'pdo_mysql',
'host' => $connectionConfig['host'],
// IT'S NOT USERNAME, IT'S USER ....
'username' => $connectionConfig['user'],
'password' => $connectionConfig['password'],
'dbname' => $connectionConfig['db_name']
];
In the StackOverflow post that is used to create this factory, it was written username, but the correct field was user.
That was my first mistake, my second mistake is that, when i tried to execute my nativeQuery, I created a resultSetMapping empty :
$entityManager = $this->factory->createManager($databaseGroup->getIdDb());
// I did this
$rsm = new ResultSetMapping();
$test = $entityManager->createNativeQuery("USE mydatabase; SELECT * FROM mytable", $rsm)->execute();
// I SHOULD HAVE DONE THIS
$rsm = new ResultSetMappingBuilder($entityManager);
$rsm->addScalarResult('id', 'id');
[... for every field]
$result = $entityManager->createNativeQuery("SELECT id, prenom, nom FROM mytable",$rsm)->execute();
I use addScalarResult because what i get from those databases are not Entities I will keep in my program.
I hope if someone get stuck like me, this could help him/her/etc..
Summary: PERN app needs to download XLSX that represents the same query that was run to populate a data grid on the client side. For example a user has defined a series of product family prefixes and they're sent to the server endpoint:
const getItemsByFamily = async (like) => {
const items = await fetch('/item/like/' + like);
const result = await items.json();
return result;
};
Where the endpoint uses a call to the associated database function:
app.get('/item/like/:like', async (req, res) => {
const { like } = req.params;
const { rows } = await dbo.getByItemPrefix(like);
res.json(rows);
});
And, the function runs the actual query against the PG Db:
getByItemPrefix: async (like) => await pool.query(`select * from myschema.items where ${like}`);
All is well... But, now stakeholders on this project have requested the ability to download a representative XLSX for the same data set. My idea is to use PG's COPY command to write a unique XLSX on the server and link to it somehow.
I start with a proof of concept, hard-coded address, just to simply download a sample xlsx:
// TODO: Add some sort of reference to represent data-grid query.
const DownloadXLSX = ({/* query or query ref */}) => {
return (
<a href='http://localhost:1234/download?fileName=sample.xlsx' target='_blank' rel='noopener noreferrer' download>
<Button
variant='contained'
size='small'
>
Download
</Button>
</a>
);
};
But, now I'm on SO because I am stuck in an architecture question. What is possible?
Is it time to audit every query and reference it somehow that way? Currently this product has no auditing, but perhaps it is time to spend time to do that? Possibly make an audit entry and send the id down with the results, passing the audit.id down do the download button for its query param?
Or, is there some more simple way to track a data grid's representative query on the client?
I like to do whatever is optimal and I'm hoping to find opinions or some insight. Thanks in advance for any help in getting over this architecture hurdle (missing having a team!).
I thought I should follow up here.
To access the actual text of a query within a PG stored function that could be used with a trigger you can use the function current_query().
However, my whole idea of using triggers went down the drain when I found this in the PG docs:
SELECT does not modify any rows so you cannot create SELECT triggers. Rules and views are more appropriate in such cases.
While the suggested approach might work in some cases it really would not for me as the results of some queries are huge and I really needed a reference.
Instead I wrap pool.query with a function that just takes the passed-in SQL and stores it in a simple, but sufficient for my purposes, "audit" table.
CREATE TABLE myschema.audit
(
id serial,
user_id varchar(50) not null,
full_query text not null,
x_time timestamp not null default now(),
CONSTRAINT audit_pkey PRIMARY KEY (id)
)
TABLESPACE pg_default;
ALTER TABLE myschema.audit
OWNER to postgres;
pool.queryWithAudit = async query => {
const { rows } = await pool.query(SQL`insert into iol.trac_audit (user_id, full_query) values ($1, $2) returning id`, ['dev-trac', query]);
const auditID = rows[0].id;
const queryResult = await pool.query(query);
queryResult.auditID = auditID;
return queryResult;
};
And, finally, the retun from the server changed just a little bit to include the auditID, and the download component had to change (to CSV for now, instead of XLSX):
const DownloadXLSX = ({ queryId, style = {} }) => {
return (
<a style={{ textDecoration: 'none', ...style }} href={`http://localhost:1234/download?queryId=${queryId}`} target='_blank' rel='noopener noreferrer' download>
<Button
endIcon={<GetAppIcon />}
size='small'
>
Download
</Button>
</a>
);
};
Along with corresponding route for lookup of query, then execution:
app.get('/download', async (req, res) => {
const { queryId } = req.query;
// Now grab query from audit and run that.
const fileName = await dbo.makeXLSX(queryId);
res.download(fileName, fileName.split('/').pop(), (err) => {
console.log(err ? 'Error attempting to download: ' + err.message : 'No server-side error in attempt to download.');
if (!err) fs.unlink(fileName, (unlinkErr) => console.log(unlinkErr ? 'Temp folder cleanup error: ' + unlinkErr.message : `File at: [${fileName}] removed from system temp folder.`));
});
});
dbo.js updates:
makeXLSX: async auditId => {
const { rows } = await pool.query(`select full_query from myschema.audit where id = ${auditId}`);
const query = rows[0].full_query;
const fileName = path.join('C:/tmp/', Date.now().toString() + '.csv');
const copyCommand = `COPY (${query}) TO '${fileName}' CSV HEADER;`;
await pool.query(copyCommand);
return fileName;
}
Anyway, this works for me. Maybe somebody will submit improvements? I just include the button if there is a queryId, which I null-out with the associated useState setter function:
const [queryId, setAuditIdRef] = useState(null);
{queryId ? <DownloadXLSX queryId={queryId} /> : null}
Good day.
I am trying to copy my table data from MSSQL server to elasticsearch, after going through some documents i wrote my config file for logstash, when i run the file from command prompt , i get this message :
"J_3a_.ELK.logstash_minus_6_dot_6_dot_2.logstash_minus_core.lib.logstash.pipeline.block
in start_input"
I have the MSSQL database in aws-RDS.
This is my conf file
input {
jdbc {
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_driver_library => "J:\Java\sqljdbc_4.2\enu\jre8\sqljdbc42.jar"
jdbc_connection_string =>
"jdbc:sqlserver://skunestdevdb.czdnys4mper2.ap-southeast-
2.rds.amazonaws.com;database=skudevusr;user=
<my_username>;password=<my_password>"
jdbc_user => "<my_username>"
jdbc_password => "<myPassword>#"
statement => "select * from product"
}
}
output{
elasticsearch{
hosts => ["localhost:9200"]
index => ["skunest_qa"]
}
}
I cant seem to get what is wrong here, i have provided necessary details in the conf file, the command line stucks showing above message. I have tried removing the username and password from the connection string.
Help!!
Try this:
input {
jdbc {
# SqlServer jdbc connection string to your database, productdb
# "jdbc:sqlserver://HostName\instanceName;database=DBName;user=UserName;password=Password"
jdbc_connection_string => "jdbc:sqlserver://localhost\SQLExpress;database=productdb;user=sa;password=test#123"
# The user you want to execute your statement as
jdbc_user => nil
# The path to your downloaded jdbc driver
jdbc_driver_library => "C:/Program Files/sqljdbc_6.0/enu/jre8/sqljdbc42.jar"
# The name of the driver class for SqlServer
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
# Query for testing purpose
statement => "SELECT * from product"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "logstash-%{+YYYY.MM.dd}"
}
}
My data is stored in Elasticsearch and I need a way to pull it and use SPSS with.
Is there any method for this? Maybe a connector layer of programming I need to do on SPSS?
Thanks in adv
The easiest way I see is to use Logstash to export your data from ES into a CSV file and then import that CSV into SPSS.
A sample Logstash config file to export your data would look like this:
input {
elasticsearch {
hosts => ["localhost:9200"]
query => '{"query":{"match_all":{}}'
index => "my-index"
scroll => "5m"
}
}
filter {
mutate {
remove_field => [ "#version", "#timestamp" ]
}
}
output {
csv {
fields => ["header1", "header2", "header3"]
path => "my-index.csv"
}
}
I am trying create ArrayDataProvider. I am calling dataProvider like this:
$dataProvider = new ArrayDataProvider([
'allModels' => $query->from('posts')->all(),
'pagination' => [
'pageSize' => 10,
],
]);
Here I am using pageSize. When I run this code in sql not showing LIMIT, but pagination is working. I have more than 100 posts and all of them loading in all time. Generated SQL Query: SELECT * FROMposts.
What I must do to solve this.
Thank you!
Try this
'pageSizeLimit' => [5,100],