How to find and hard delete objects older than n-days in LakeFS? Later it'll be a scheduled job.
To do that you should use the Garbage Collection (GC) feature in lakeFS.
Note: This feature cleans objects from the storage only after they are deleted from your branches in lakeFS.
You will need to:
Define GC rules to set your desired retention period.
From the lakeFS UI, go to the repository you would like to hard delete objects from -> Settings -> Retention, and define the GC rule for each branch under the repository. For example -
{
"default_retention_days": 21,
"branches": [
{"branch_id": "main", "retention_days": 28},
{"branch_id": "dev", "retention_days": 7}
]
}
Run the GC Spark job that does the actual cleanup, with -
spark-submit --class io.treeverse.clients.GarbageCollector \
-c spark.hadoop.lakefs.api.url=https://lakefs.example.com:8000/api/v1 \
-c spark.hadoop.lakefs.api.access_key=<LAKEFS_ACCESS_KEY> \
-c spark.hadoop.lakefs.api.secret_key=<LAKEFS_SECRET_KEY> \
-c spark.hadoop.fs.s3a.access.key=<S3_ACCESS_KEY> \
-c spark.hadoop.fs.s3a.secret.key=<S3_SECRET_KEY> \
--packages io.lakefs:lakefs-spark-client-301_2.12:0.5.0 \
example-repo us-east-1
Related
I've downloaded ServiceMix 7.0.1 and followed the documentation. I need to install Camel features using:
karaf#root>feature:install camel-sql
But when I do that nothing happens. There's no response until I hit ctrl-c. The log file doesn't help either.
2020-03-25 14:13:25,709 | INFO | Thread-16 | FeaturesServiceImpl
| 8 - org.apache.karaf.features.core - 4.0.9 | Adding features: camel-sql/[2.24.2,2.24.2]
And the online documentation is sparse to say the least.
Update
Using -v -t gives the below output, but still nothing thereafter:
karaf#root>feature:install -v -t camel-sql
Adding features: camel-sql/[2.24.2,2.24.2]
You have to update Maven configuration in etc/org.ops4j.pax.url.mvn.cfg in your ServiceMIX 7.0.1 installation. Change:
org.ops4j.pax.url.mvn.repositories= \
http://repo1.maven.org/maven2#id=central, \
http://repository.springsource.com/maven/bundles/release#id=spring.ebr.release, \
http://repository.springsource.com/maven/bundles/external#id=spring.ebr.external, \
http://zodiac.springsource.com/maven/bundles/release#id=gemini, \
http://repository.apache.org/content/groups/snapshots-group#id=apache#snapshots#noreleases, \
https://oss.sonatype.org/content/repositories/snapshots#id=sonatype.snapshots.deploy#snapshots#noreleases, \
https://oss.sonatype.org/content/repositories/ops4j-snapshots#id=ops4j.sonatype.snapshots.deploy#snapshots#noreleases
to:
org.ops4j.pax.url.mvn.repositories= \
https://repo1.maven.org/maven2#id=central, \
http://repository.springsource.com/maven/bundles/release#id=spring.ebr.release, \
http://repository.springsource.com/maven/bundles/external#id=spring.ebr.external, \
http://zodiac.springsource.com/maven/bundles/release#id=gemini, \
http://repository.apache.org/content/groups/snapshots-group#id=apache#snapshots#noreleases, \
https://oss.sonatype.org/content/repositories/snapshots#id=sonatype.snapshots.deploy#snapshots#noreleases, \
https://oss.sonatype.org/content/repositories/ops4j-snapshots#id=ops4j.sonatype.snapshots.deploy#snapshots#noreleases
repo1.maven.org no longer allows access using http. https is required.
Can you please try bundle:install mvn:org.apache.camel/camel-sql/2.24.2
I am working on yocto, I want to compile some C files in yocto and install the resulting binary to external filesystem.
Before doing that I tried creating a separate reciepe and compile c code from it.
I am unable to compile it.
I am not sure to understand the question since it is not precise enough.
Including C files in recipe tree
If you want to have the C files in your recipe, having a file tree like this:
recipe-example/example/example_0.1.bb
recipe-example/example/example-0.1/helloworld.c
You can generate this example when you create a new layer using
yocto-layer <your-layer-name>
Your bb file will look like this:
#
# This file was derived from the 'Hello World!' example recipe in the
# Yocto Project Development Manual.
#
SUMMARY = "Simple helloworld application"
SECTION = "examples"
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
SRC_URI = "file://helloworld.c"
S = "${WORKDIR}"
do_compile() {
${CC} helloworld.c -o helloworld
}
do_install() {
install -d ${D}${bindir}
install -m 0755 helloworld ${D}${bindir}
}
It will compile the hello world file and install it into /usr/bin on your image.
From a Git repo
You also can compile from a git repository, I advise you to read the manual and examples in your yocto folder. Here is an example here of wiringPi:
DESCRIPTION = "A library to control Raspberry Pi GPIO channels"
HOMEPAGE = "https://projects.drogon.net/raspberry-pi/wiringpi/"
SECTION = "devel/libs"
LICENSE = "LGPLv3+"
LIC_FILES_CHKSUM = "file://COPYING.LESSER;md5=e6a600fd5e1d9cbde2d983680233ad02"
# tag 2.29
SRCREV = "d79506694d7ba1c3da865d095238289d6175057d"
S = "${WORKDIR}/git"
SRC_URI = "git://git.drogon.net/wiringPi \
file://0001-Add-initial-cross-compile-support.patch \
file://0001-include-asm-ioctl.h-directly-for-_IOC_SIZEBITS.patch \
"
COMPATIBLE_MACHINE = "raspberrypi"
CFLAGS_prepend = "-I${S}/wiringPi -I${S}/devLib"
EXTRA_OEMAKE += "'INCLUDE_DIR=${D}${includedir}' 'LIB_DIR=${D}${libdir}'"
EXTRA_OEMAKE += "'DESTDIR=${D}/usr' 'PREFIX=""'"
do_compile() {
oe_runmake -C devLib
oe_runmake -C wiringPi
oe_runmake -C gpio 'LDFLAGS=${LDFLAGS} -L${S}/wiringPi -L${S}/devLib'
}
do_install() {
oe_runmake -C devLib install
oe_runmake -C wiringPi install
oe_runmake -C gpio install
}
It is fetching from a git repository, applying patches generated by git, using oe_runmake to compile with the makefiles.
With devtool
It has been asked in a comment on how to add a recipe with devtool.
We will still use wiringPi as an example again. Download it doing
https://github.com/WiringPi/WiringPi
The Makefile is is the folder wiringPi.
You can then do
devtool add <name_of_recipe> <path_to_Makefile_folder>
Take care of the warning from devtool
NOTE: Creating workspace layer in /home/dbensoussan/new_poky/poky/build/workspace
NOTE: Enabling workspace layer in bblayers.conf
NOTE: Using source tree as build directory since that would be the default for this recipe
NOTE: Recipe /home/dbensoussan/new_poky/poky/build/workspace/recipes/project/project.bb has been automatically created; further editing may be required to make it fully functional
This is generating the recipe as follow:
# Recipe created by recipetool
# This is the basis of a recipe and may need further editing in order to be fully functional.
# (Feel free to remove these comments when editing.)
#
# WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is
# your responsibility to verify that the values are complete and correct.
LICENSE = "Unknown"
LIC_FILES_CHKSUM = "file://COPYING.LESSER;md5=e6a600fd5e1d9cbde2d983680233ad02"
# No information for SRC_URI yet (only an external source tree was specified)
SRC_URI = ""
# NOTE: this is a Makefile-only piece of software, so we cannot generate much of the
# recipe automatically - you will need to examine the Makefile yourself and ensure
# that the appropriate arguments are passed in.
do_configure () {
# Specify any needed configure commands here
:
}
do_compile () {
# You will almost certainly need to add additional arguments here
oe_runmake
}
do_install () {
# This is a guess; additional arguments may be required
oe_runmake install 'DESTDIR=${D}'
}
You can then edit your recipe to suit your configuration
With externalsrc
It is possible to use a directory present on the filesystem by using externalsrc.
I did not try it myself, nor have I the workspace ready to do, but #71GA in the comment tested the tutorial from the Koan software company https://wiki.koansoftware.com/index.php/Building_Software_from_an_External_Source and it worked. I will copy the content here:
in this case use the externalsrc class - you can inherit this in the original bb recipe or a bbappend:
inherit externalsrc
EXTERNALSRC = "/path/to/sources"
Depending on the type of build (eg, 'inherit module' for out of tree Linux kernel modules) you may or may not need to set EXTERNALSRC_BUILD.
inherit externalsrc
EXTERNALSRC = "/some/path"
EXTERNALSRC_BUILD = "/some/path"
If you're going to use it across a number of recipes you can inherit it globally at the configuration level (perhaps via an inc file that you include/require there):
INHERIT += "externalsrc"
EXTERNALSRC_pn-<recipename> = "/path/to/sources"
Recipe example using an external source for nInvaders package
#
# Recipe example with externalsrc
#
# (C)2019 Marco Cavallini - KOAN - <https://koansoftware.com>
#
LICENSE = "CLOSED"
LIC_FILES_CHKSUM = ""
inherit externalsrc
EXTERNALSRC = "/home/koan/yocto-qemuarm-sumo/ninvaders-0.1.1"
EXTERNALSRC_BUILD = "${EXTERNALSRC}"
DEPENDS = "ncurses"
EXTRA_OEMAKE = "-e"
do_install() {
install -d ${D}${bindir}
install -m 0755 nInvaders ${D}${bindir}
}
FILES_${PN} = "${bindir}/*"
You can just go to the official documentation to find your answer.
In the chapter 7.3 Writting a New Recipe of the yocto mega-manual, the very first example is exactly is what you need (Single .c File Package (Hello World!)).
There is already some functionality in tensorflow to create benchmarks which can be seen in action for example in the adjust contrast op benchmark. If I run this on my machine, however, I just get an empty output:
panmari#dingle:~/tensorflow$ bazel run //tensorflow/core:kernels_adjust_contrast_op_benchmark_test --test_output=all --cache_test_results=no -- --benchmarks=1000
INFO: Found 1 target...
Target //tensorflow/core:kernels_adjust_contrast_op_benchmark_test up-to-date:
bazel-bin/tensorflow/core/kernels_adjust_contrast_op_benchmark_test
INFO: Elapsed time: 10.736s, Critical Path: 8.71s.
INFO: Running command line: bazel-bin/tensorflow/core/kernels_adjust_contrast_op_benchmark_test '--benchmarks=1000'.
Running main() from test_main.cc
Benchmark Time(ns) Iterations
--------------------------------
Is my invocation of wrong?
To invoke the benchmarks, run the following command (passing --benchmarks=all as the final argument):
$ bazel run -c opt //tensorflow/core:kernels_adjust_contrast_op_benchmark_test \
--test_output=all --cache_test_results=no -- --benchmarks=all
To run GPU benchmarks, you must pass --config=cuda to bazel and append _gpu to the name of the test target. For example:
$ bazel run -c opt --config=cuda \
//tensorflow/core:kernels_adjust_contrast_op_benchmark_test_gpu \
--test_output=all --cache_test_results=no -- --benchmarks=all
Main goal
I need openSSL. Apple deprecated this library since 10.7 It is even impossible to create context when ignoring warnings about deprecation.
I decided to build openSSL and tweak it to my needs:
disable SSL protocol (only TLS should be supported)
limit cipher suites to secure one
What I have done
How I'm trying to build it.
I've downloaded sources version 1.0.2d and configured it this way:
./Configure -openssldir=$(pwd)/output-build \
enable-ec_nistp_64_gcc_128 \
no-ssl2 no-ssl3 \
no-shared no-comp \
enable-TLS_DHE_RSA_WITH_AES_256_CBC_SHA \
enable-TLS_DH_RSA_WITH_AES_256_CBC_SHA \
enable-TLS_DHE_DSS_WITH_AES_256_CBC_SHA \
enable-TLS_DH_DSS_WITH_AES_256_CBC_SHA \
enable-TLS_RSA_WITH_AES_256_CBC_SHA \
enable-TLS_DHE_RSA_WITH_AES_128_CBC_SHA \
enable-TLS_DH_RSA_WITH_AES_128_CBC_SHA \
enable-TLS_DHE_DSS_WITH_AES_128_CBC_SHA \
enable-TLS_DH_DSS_WITH_AES_128_CBC_SHA \
enable-TLS_RSA_WITH_AES_128_CBC_SHA \
enable-TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA \
enable-TLS_RSA_WITH_3DES_EDE_CBC_SHA \
darwin64-x86_64-cc
Problem
Switches for disabling SSL are clear and I'm happy with it.
Switches for cipher suites are very inconvenient, I'm not sure how they work. They are not documented, ./Configure --help prints this:
Usage: Configure [no-<cipher> ...] [enable-<cipher> ...]
[experimental-<cipher> ...] [-Dxxx] [-lxxx] [-Lxxx] [-fxxx] [-Kxxx]
[no-hw-xxx|no-hw] [[no-]threads] [[no-]shared]
[[no-]zlib|zlib-dynamic] [no-asm] [no-dso] [no-krb5] [sctp] [386]
[--prefix=DIR] [--openssldir=OPENSSLDIR] [--with-xxx[=vvv]]
[--test-sanity] os/compiler[:flags]
I just want listed cipher suites and nothing else. Format of this switches give me impression that I have to disable quite long list of other cipher suites. Is there better way so I can disabled all but those listed above?
I want all my modules built with a .so extension so that they are shared. However, all my modules are building with a .la extension. Here are the contents of the module directory:
httpd.exp libmod_proxy_ajp.a libmod_speling.a mod_info.la mod_proxy_scgi.la
libmod_cern_meta.a libmod_proxy_balancer.a libmod_ssl.a mod_proxy.la mod_rewrite.la
libmod_dbd.a libmod_proxy_connect.a libmod_vhost_alias.a mod_proxy_ajp.la mod_speling.la
libmod_deflate.a libmod_proxy_ftp.a mod_cern_meta.la mod_proxy_balancer.la mod_ssl.la
libmod_expires.a libmod_proxy_http.a mod_dbd.la mod_proxy_connect.la mod_vhost_alias.la
libmod_info.a libmod_proxy_scgi.a mod_deflate.la mod_proxy_ftp.la
libmod_proxy.a libmod_rewrite.a mod_expires.la mod_proxy_http.la
Here is my configure command:
./configure \
--with-mpm=prefork \
--with-ssl=/opt/freeware \
--enable-cern-meta \
--enable-dbd \
--enable-deflate \
--enable-expires \
--enable-info \
--enable-proxy \
--enable-rewrite \
--enable-speling \
--enable-ssl \
--enable-vhost-alias \
--enable-mods-shared="cern_meta dbd deflate expires info proxy rewrite speling ssl vhost_alias"
I have done this many times in the past on AIX 5.2, 5.3 and 6.1 without issue.
.la files are libtool's library control file syntax. Notice they are really just small scripts. Check for your shared library objects inside the .libs directory. You might also explicitly tell configure that you want shared libraries with --enable-shared