Shake: depend on rules from Shakefile in subdirectory? - shake-build-system

I've got a project that looks something like this:
.
├── pyproject.toml
├── server.py
├── Shakefile.hs
└── ui
├── index.ts
└── Shakefile.hs
ui/Shakefile.hs contains something to the effect of:
import Development.Shake
main = shakeArgs shakeOptions $ do
want ["dist/index.js"]
"dist/index.js" %> \out -> do
need ["index.ts"]
cmd "tsc --outFile" [out]
and the root Shakefile.hs looks like:
import Development.Shake
main = shakeArgs shakeOptions $ do
want ["dist/server.tar"]
"dist/server.tar" %> \out -> do
need ["ui/dist/index.js", "server.py"]
cmd "tar cf" [out, "ui/dist/index.js", "server.py"]
Question: how should I model dist/server.tar's dependency on ui/dist/index.js?
My first guess was simply to add a rule specifying how to build ui/dist/index.html relative to the project root, with the action being to simply call shake (so that I don't have to keep the recipes in sync):
-- added to ./Shakefile.hs
"ui/dist/index.js" %> \_ -> do
cmd "stack exec shake -- -C ui dist/index.js"
BUT, considering the Shake paper cites Recursive Make Considered Harmful, I'm near certain this isn't the way to go.
Is there a way to import rules from another Shakefile? Can vanilla Haskell's import system be used to implement something like this? In either case, how could we (automatically?) translate rules and actions using paths relative to the child directory to be correct relative to the parent?
In general, I'm wondering about Shake's applicability to monorepos, and about the composability of different Shake-based build systems (i.e. Shakefiles from different internal projects).
Thanks for your consideration!

Related

Sonar scanner runs more sub-modules than needed

When I run sonar-scanner on C project, it analyzes my submodules, but it also includes a "parent" configuration. What I mean:
sonar-project.properties
sonar.host.url=<host>
sonar.login=<token>
sonar.projectKey=PROJECT1
sonar.qualitygate.wait=true
sonar.sourceEncoding=UTF-8
sonar.scm.provider=git
sonar.verbose=true
sonar.modules=module1,module2
sonar.language=C
# if I enable below 3 lines it won't crash
#sonar.sources=.
#sonar.cfamily.build-wrapper-output=code_coverage_results
#sonar.cfamily.gcov.reportsPath=code_coverage_results
module1.sonar.projectName=Module1
module1.sonar.projectBaseDir=./module1
module1.sonar.sources=src
module1.sonar.tests=tests
module1.sonar.cfamily.build-wrapper-output=../code_coverage_results
module1.sonar.cfamily.gcov.reportsPath=../code_coverage_results
module1.sonar.exclusions=**/*.py,src/http-parser/**,obj/**,bin/**
module2.sonar.projectName=Module2
module2.sonar.projectBaseDir=./module2
module2.sonar.sources=src
module2.sonar.tests=tests
module2.sonar.cfamily.build-wrapper-output=../code_coverage_results
module2.sonar.cfamily.gcov.reportsPath=../code_coverage_results
sonar.cfamily.threads=10
sonar.cfamily.cache.enabled=true
sonar.cfamily.cache.path=sonar.cfamily.cache
sonar.scm.exclusions.disabled=true
code_coverage_results is a common folder for both submodules, basically can be separated.
What I see when run sonar-scanner -X:
------------- Run sensors on module Module1
... <analyzing> ...
------------- Run sensors on module Module2
... <analyzing> ...
------------- Run sensors on module PROJECT1
17:39:47.092 INFO: Sensor CFamily [cpp]
17:39:47.093 INFO: CFamily plugin version: 6.32.0.44918
17:39:47.093 ERROR:
he only way to get an accurate analysis of C/C++/Objective-C files is by using the SonarSource build-wrapper and setting the property "sonar.cfamily.build-wrapper-output" or by using Clang Compilation Database and setting the property "sonar.cfamily.compile-commands". None of these two options were specified.
If you don't want to analyze C/C++/Objective-C files, then prevent them from being analyzed by setting the following properties:
sonar.c.file.suffixes=-
sonar.cpp.file.suffixes=-
sonar.objc.file.suffixes=-
at com.sonar.cpp.plugin.CFamilySensor.process(CFamilySensor.java:236)
at com.sonar.cpp.plugin.CFamilySensor.execute(CFamilySensor.java:186)
at org.sonar.scanner.sensor.AbstractSensorWrapper.analyse(AbstractSensorWrapper.java:64)
at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:85)
at org.sonar.scanner.sensor.ModuleSensorsExecutor.execute(ModuleSensorsExecutor.java:62)
at org.sonar.scanner.scan.SpringModuleScanContainer.doAfterStart(SpringModuleScanContainer.java:81)
at org.sonar.core.platform.SpringComponentContainer.startComponents(SpringComponentContainer.java:188)
at org.sonar.core.platform.SpringComponentContainer.execute(SpringComponentContainer.java:167)
at org.sonar.scanner.scan.SpringProjectScanContainer.scan(SpringProjectScanContainer.java:392)
When I uncomment the lines:
sonar.sources=.
sonar.cfamily.build-wrapper-output=code_coverage_results
sonar.cfamily.gcov.reportsPath=code_coverage_results
I am getting the error like:
File module2/src/Source.c can't be indexed twice. Please check that inclusion/exclusion patterns produce disjoint sets for main and test files
That is why I have to put
sonar.exclusions=**
in the beginning.
The question is: why it tries to analyze three modules instead of two?
I have the next structure:
root folder
Module1
Module2
CommonSources
*.sh, *.py files
sonar-project.properties
Thanks.

How to add additional files in Sagemaker Pipeline Processing Step

I want to have additional files which can be imported in preprocess.py file
but i am not able to import these directly.
My directory looks like this:
Want to import from helper_functions directory into preprocess.
I tried to add this in setup.py file but it did not work.
package_data={"pipelines.ha_forecast.helper_functions": ["*.py"]},
One thing which kind of worked was to add this folder in input like this:
inputs = [
ProcessingInput(source=f'{project_name}/{module_name}/helper_functions',
destination="/opt/ml/processing/input/code/helper_functions"),
]
But this was putting the required files in some other directory which I was not able to import anymore.
What is standard way of doing this?
You have to specify the source_dir. Within your script then you can import the modules as you normally do.
source_dir (str or PipelineVariable) – Path (absolute, relative or an
S3 URI) to a directory with any other training source code
dependencies aside from the entry point file (default: None). If
source_dir is an S3 URI, it must point to a tar.gz file. Structure
within this directory are preserved when training on Amazon SageMaker.
Look at the documentation in general for Processing (you have to use FrameworkProcessor and not the specific ones like SKLearnProcessor).
P.S.: The answer is similar to that of the question "How to install additional packages in sagemaker pipeline".
Within the specified folder, there must be the script (in your case preprocess.py), any other files/modules that may be needed, and also eventually the requirements.txt file.
The structure of the folder then will be:
BASE_DIR/
|─ helper_functions/
| |─ your_utils.py
|─ requirements.txt
|─ preprocess.py
Within your preprocess.py, you will call the scripts in a simple way with:
from helper_functions.your_utils import your_class, your_func
So, your code becomes:
from sagemaker.processing import FrameworkProcessor
from sagemaker.sklearn import SKLearn
from sagemaker.workflow.steps import ProcessingStep
from sagemaker.processing import ProcessingInput, ProcessingOutput
BASE_DIR = your_script_dir_path
sklearn_processor = FrameworkProcessor(
estimator_cls=SKLearn,
framework_version=framework_version,
instance_type=processing_instance_type,
instance_count=processing_instance_count,
base_job_name=base_job_name,
sagemaker_session=pipeline_session,
role=role
)
step_args = sklearn_processor.run(
inputs=[your_inputs],
outputs=[your_outputs],
code="preprocess.py",
source_dir=BASE_DIR,
arguments=[your_arguments],
)
step_process = ProcessingStep(
name="ProcessingName",
step_args=step_args
)
It's a good practice to keep the folders for the various steps separate for each and don't create overlaps.

How to install additional packages in sagemaker pipeline

I want to add dependency packages in my sagemaker pipeline which will be used in Preprocess step.
I have tried to add it in required_packages in setup.py file but it's not working.
I think setup.py file is no use of at all.
required_packages = ["sagemaker==2.93.0", "matplotlib"]
Preprocessing steps:
sklearn_processor = SKLearnProcessor(
framework_version="0.23-1",
instance_type=processing_instance_type,
instance_count=processing_instance_count,
base_job_name=f"{base_job_prefix}/job-name",
sagemaker_session=pipeline_session,
role=role,
)
step_args = sklearn_processor.run(
outputs=[
ProcessingOutput(output_name="train", source="/opt/ml/processing/train"),
ProcessingOutput(output_name="validation", source="/opt/ml/processing/validation"),
ProcessingOutput(output_name="test", source="/opt/ml/processing/test"),
],
code=os.path.join(BASE_DIR, "preprocess.py"),
arguments=["--input-data", input_data],
)
step_process = ProcessingStep(
name="PreprocessSidData",
step_args=step_args,
)
Pipeline definition:
pipeline = Pipeline(
name=pipeline_name,
parameters=[
processing_instance_type,
processing_instance_count,
training_instance_type,
model_approval_status,
input_data,
],
steps=[step_process],
sagemaker_session=pipeline_session,
)
For each job within the pipeline you should have separate requirements (so you install only the stuff you need in each step and have full control over it).
To do this, you need to use the source_dir parameter:
source_dir (str or PipelineVariable) – Path (absolute, relative or an
S3 URI) to a directory with any other training source code
dependencies aside from the entry point file (default: None). If
source_dir is an S3 URI, it must point to a tar.gz file. Structure
within this directory are preserved when training on Amazon SageMaker.
Look at the documentation in general for Processing (you have to use FrameworkProcessor).
Within the specified folder, there must be the script (in your case preprocess.py), any other files/modules that may be needed, and the requirements.txt file.
The structure of the folder then will be:
BASE_DIR/
|- requirements.txt
|- preprocess.py
It is the common requirements file, nothing different. And it will be used automatically at the start of the instance, without any instruction needed.
So, your code becomes:
from sagemaker.processing import FrameworkProcessor
from sagemaker.sklearn import SKLearn
from sagemaker.workflow.steps import ProcessingStep
from sagemaker.processing import ProcessingInput, ProcessingOutput
sklearn_processor = FrameworkProcessor(
estimator_cls=SKLearn,
framework_version='0.23-1',
instance_type=processing_instance_type,
instance_count=processing_instance_count,
base_job_name=f"{base_job_prefix}/job-name",
sagemaker_session=pipeline_session,
role=role
)
step_args = sklearn_processor.run(
outputs=[
ProcessingOutput(output_name="train", source="/opt/ml/processing/train"),
ProcessingOutput(output_name="validation", source="/opt/ml/processing/validation"),
ProcessingOutput(output_name="test", source="/opt/ml/processing/test"),
],
code="preprocess.py",
source_dir=BASE_DIR,
arguments=["--input-data", input_data],
)
step_process = ProcessingStep(
name="PreprocessSidData",
step_args=step_args
)
Note that I changed both the code parameter and the source_dir. It's a good practice to keep the folders for the various steps separate so you have a requirements.txt for each and don't create overlaps.

How to make a unique layout for the main page?

I have multiple pages on my web-site and only main page should have a different layout? How can I achieve this?
src/
- page1/ <- LayoutB
- page2/ <- LayoutB
- page3/ <- LayoutB
- index.svelte <- LayoutA
If I use __layout.svelte in the src/ folder it will apply for every page.
Update November 2022
This approach was tested in
npm -v #sveltejs/kit
8.19.2
To make a unique layout for a specific page (or group of pages) you need to use group.
src/
routes/
(unauthenticated)/
welcome/
register/
login/
+page.svelte
+layout.svelte (this layout applies to welcome/, register/ and login/
(authenicated)/
home/
settings/
+page.svelte
+layout.svelte (this layout applies to home/, settings/ and dashboard/
+page.svelte
+layout.svelte (this layout applies to all pages in unauthenticated/ and authenticated/)
I was able to find a solution by using named layouts from SvelteKit
The solution to my problem will look like this
src/
- page1/ <- LayoutB
- page2/ <- LayoutB
- page3/ <- LayoutB
- __layout.svelte (LayoutB)
- __layout-main.svelte (LayoutA)
- index#main.svelte <- LayoutA
I created a layout file with -main suffix and used this name as a reference to the layout in my index#main.svelte file

PlantUML not finding custom stylesheets

I attempted to replicate the directory structure in this tutorial by jerieljan. Unfortunately I seem to be getting the default PlantUML stylings. I have included the expected results vs. the actual results in screenshots below. I am using the Visual Studio Code extension by jebbs with preview. While the style sheets are too long to include here they can be found in the aforementioned linked tutorial. The diagram file is as follows:
# Sequence Diagram Example
#startuml
'General Defaults
!define BASEPATH ..
!include BASEPATH/globals/stylesheet.iuml
'Actor Definitions
autonumber 1 "0:"
title Sequence Diagram
A -> B: Perform Action
B --> A: (Response)
#enduml
The directory structure is the same as in the tutorial and is as follows:
$ tree
.
├── globals
│ ├── style-presets.iuml
│ └── stylesheet.iuml
└── diagrams
├── example-sequence.puml
├── etc...
Expected Diagram
Actual Diagram

Resources